Adaptive throttling in a universal backup host

ABSTRACT

Systems and methods to throttle a universal backup host are described. The system executes a job, at a backup host, to back up a file set from a source host including fetching metadata from the source host. The system identifies a first operation set from operation sets, the operation set including a first operation. The system communicates, in parallel, requests for metadata items, over a network, to the source host, receives responses, and processes the responses by utilizing threads from a thread pool. The system generates latencies, counts the number of requests, and stores the latencies and number of requests in samples. The system aggregates the samples responsive to a timeout. The system resizes the thread pool based on the aggregating. Finally, the system backs up the file set from the source host based on the metadata.

TECHNICAL FIELD

The disclosure relates generally to computer architecture software for adata management platform and, in some more particular aspects, toadaptive throttling in a universal backup host.

BACKGROUND

Enterprise resource planning (ERP) systems, customer resource management(CRM) systems, and other production systems require repeated recovery,testing, and analysis. Accordingly, such systems are frequently backedup. But frequent backup of production systems may take a prohibitiveamount of time and/or burden the production system with processes thatcompete for scarce resources.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram illustrating a system, according to anembodiment, to utilize adaptive throttling;

FIG. 1B is a block diagram illustrating a system, according to anembodiment, to utilize adaptive throttling;

FIG. 1C is a block diagram illustrating a system, according to anembodiment, to utilize adaptive throttling;

FIG. 1D is a block diagram illustrating a moving average, according toan embodiment;

FIG. 1E is a block diagram illustrating a moving average, according toan embodiment;

FIG. 1F is a block diagram illustrating a moving average, according toan embodiment;

FIG. 1G is a block diagram illustrating a condition, according to anembodiment, to increase a size of a thread pool;

FIG. 1H is a block diagram illustrating a condition, according to anembodiment, to decrease the size of the thread pool;

FIG. 2A is a block diagram illustrating a thread pool, according to anembodiment;

FIG. 2B is a block diagram illustrating a thread, according to anembodiment;

FIG. 2C is a block diagram illustrating thread pool information,according to an embodiment;

FIG. 2D is a block diagram illustrating a moving average repository,according to an embodiment;

FIG. 2E is a block diagram illustrating a moving average, according toan embodiment;

FIG. 2F is a block diagram illustrating an initial sample, according toan embodiment;

FIG. 2G is a block diagram illustrating a moving average, according toan embodiment;

FIG. 2H is a block diagram illustrating a subsequent sample, accordingto an embodiment;

FIG. 3A is a block diagram illustrating a method, according to anembodiment, for adaptive throttling;

FIG. 3B is a block diagram illustrating a method, according to anembodiment, for communicating requests and responses;

FIG. 3C is a block diagram illustrating a method, according to anembodiment, for processing the responses;

FIG. 4 is a block diagram illustrating a method, according to anembodiment, for generating a latency;

FIG. 5 is a block diagram illustrating a method, according to anembodiment, for aggregating latencies and number of responses;

FIG. 6 is a block diagram illustrating a method, according to anembodiment, for resizing the thread pool;

FIG. 7A is a block diagram illustrating a networked computingenvironment, according to an embodiment;

FIG. 7B is a block diagram illustrating a server, according to anembodiment;

FIG. 7C is a block diagram illustrating a storage appliance, accordingto an embodiment;

FIG. 7D is a block diagram illustrating a cluster, according to anembodiment;

FIG. 8 is a block diagram illustrating a representative softwarearchitecture; and

FIG. 9 is a block diagram illustrating components of a machine,according to some example embodiments.

DETAILED DESCRIPTION

This description is directed at three aspects of a computer architecturesoftware for a data management platform. The first aspect is adaptivethrottling in a universal backup host; the second aspect is adaptivethrottling for a source host; and the third aspect is an adaptivethrottling communication protocol

Adaptive Throttling in a Universal Backup Host

According to a first aspect of the present disclosure, a system foradaptive throttling in a universal backup host is described. The systemexecutes a job, at a backup host. The job is for backing up a file setfrom a source host to the backup host. The backup host may execute thejob responsive to a triggering event. For example, the triggering eventmay include a periodic expiration of a timeout (e.g., every hour, on thehour). The backing up of the file set includes: 1) the backup hostfetching metadata from the source host; and, 2) backing up the file setfrom the remote host based on the metadata. The backup host begins thefetching of each metadata item from a remote server, at the source host,by identifying an operation set from multiple operation sets. The backuphost identifies the operation set by identifying a file sharing protocolbeing utilized by the backup host and the remote servers. For example,the backup host may identify a Unix operation set by identifying thebackup host and the remote server as utilizing a Network File Systemfile sharing protocol. The backup host selects an operation forgenerating a metadata item from the operation set. The backup hostcommunicates, in parallel, one or more requests for metadata items, overa network, to the source host by utilizing one or more threads from athread pool. For example, the backup host may communicate a first andsecond request for metadata items, in parallel, by respectivelyutilizing first and second threads from a thread pool. The first requestincludes a first operation and the second request includes a secondoperation. The backup host receives responses, over the network,corresponding to the one or more requests by using the one or morethreads. For example, the backup host receives a response from thesource host corresponding to the first request by using the firstthread. The response includes a first metadata that was generated at thesource host by executing the first operation. The backup host processesthe responses including the first response. For example, the backup hostmay process the first response by generating a first latency and storingthe first latency in a sample. The backup host may compute the firstlatency based on the elapsed time for communicating the first request,generating the metadata item on the remote host, and receiving the firstresponse. In addition, the backup host counts the first request byincrementing a number of requests in the sample. Responsive to a timeout(e.g., one second), the backup host aggregates samples. For example, thebackup host may aggregate ten one-second samples to generate oneten-second sample. The samples include the first latency and the numberof requests. The backup host resizes the thread pool based on theaggregating of the samples. Resizing the thread pool causes a throttlingof the communicating of requests. For example, a backup host with athread pool of eight may send eight requests for metadata items inparallel, but a backup host with a thread pool of four may only sendfour requests for metadata items in parallel. After the entirety of themetadata is retrieved, the backup host backs up the file set from thesource host to the backup host. The backup host backs up the file setfrom the source host based on the metadata.

Adaptive Throttling for a Source Host

According to a second aspect of the present disclosure, a system foradaptive throttling for a source host is described. The system executesa job, at a backup host. The job is for backing up a file set from thesource host to the backup host. The backup host may execute the jobresponsive to a triggering event. For example, the triggering event mayinclude a periodic expiration of a timeout (e.g., every hour, on thehour). The backing up of the file set includes: 1) the backup hostfetching metadata from the source host; and, 2) backing up the file setfrom the remote host based on the metadata. The backup host utilizes anoperation set in accordance with a file sharing protocol that is beingshared by the backup host and the source host. The backup hostcommunicates, in parallel, one or more requests for metadata items, overa network, to the source host by utilizing one or more threads from athread pool. For example, the backup host may communicate a first andsecond request for metadata items, in parallel, by respectivelyutilizing first and second threads from a thread pool. The first requestincludes a first operation and the second request includes a secondoperation. The backup host receives responses, over the network,corresponding to the one or more requests by using the one or morethreads. For example, the backup host receives a response from thesource host corresponding to the first request by using the firstthread. The response includes a first metadata item that was generated,at the source host, by executing the first operation. The backup hostprocesses the responses including the first response. For example, thebackup host may process the first response by generating a first latencyand storing the first latency in a sample. The backup host may computethe first latency based on the elapsed time for communicating the firstrequest, generating the metadata item on the remote host, and receivingthe first response. In addition, the backup host counts the firstrequest by incrementing a number of requests in the sample. Responsiveto a timeout (e.g., one second), the backup host aggregates samples. Forexample, the backup host may aggregate ten one-second samples togenerate one ten-second sample. The samples include the first latencyand the number of requests. The backup host resizes the thread poolbased on the aggregating of the samples. Resizing the thread pool causesa throttling of the communicating of requests. For example, a backuphost with a thread pool of eight may send eight requests for metadataitems in parallel, but a backup host with a thread pool of four may onlysend four requests for metadata items in parallel. After the entirety ofthe metadata is retrieved, the backup host backs up the file set fromthe source host to the backup host. The backup host backs up the fileset from the source host based on the metadata.

Adaptive Throttling Communication Protocol

According to a third aspect of the present disclosure, a systemutilizing an adaptive throttling communication protocol is described.The system includes a first host requesting one or more portions of datafrom a second host. The first host utilizes an operation set inaccordance with a file sharing protocol that is being shared by thefirst host and the second host. The first host communicates, inparallel, one or more requests for portions of data, over a network, tothe second host by utilizing one or more threads from a thread pool. Forexample, the first host may communicate a first and second request forportions of data, in parallel, by respectively utilizing first andsecond threads from a thread pool. The first request includes a firstoperation and the second request includes a second operation. The firsthost receives responses, over the network, corresponding to the one ormore requests by using the one or more threads. For example, the firsthost receives a response from the second host corresponding to the firstrequest by using the first thread. The response includes a first portionof data that was generated, at the second host, by executing the firstoperation. The first host processes the responses including the firstresponse. For example, the first host may process the first response bygenerating a first latency and storing the first latency in a sample.The first host may compute the first latency based on the elapsed timefor communicating the first request, generating the portion of data atthe remote host, and receiving the first response. In addition, thefirst host counts the first request by incrementing and storing a numberof requests in the sample. Responsive to a timeout (e.g., one second),the first host aggregates samples. For example, the first host mayaggregate ten one-second samples to generate one ten-second sample. Thesamples include the first latency and the number of requests. The firsthost resizes the thread pool based on the aggregating of the samples.Resizing the thread pool causes a throttling of the communicating ofrequests. For example, a first host with a thread pool of eight may sendeight requests for portions of data in parallel, but a first host with athread pool of four may only send four requests for portions of data inparallel.

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques,instruction sequences, and computing machine program products thatembody illustrative embodiments of the present disclosure. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofexample embodiments. It will be evident, however, to one skilled in theart that the present inventive subject matter may be practiced withoutthese specific details.

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever. The following notice applies to the software and dataas described below and in the drawings that form a part of thisdocument: Copyright Rubrik, Inc., 2019-2020, All Rights Reserved.

It will be appreciated that some of the examples disclosed herein aredescribed in the context of virtual machines that are backed up by usingbase and incremental snapshots, for example. This should not necessarilybe regarded as limiting of the disclosures. The disclosures, systems,and methods described herein apply not only to virtual machines of alltypes that run a file system (for example), but also to NAS devices,physical machines (for example Linux servers), and databases.

FIG. 1A is a block diagram illustrating a system 100, according to anembodiment, utilizing adaptive throttling. The system 100 includes anetworked system 102 including a backup host 104, a source host 106, anda client machine 111. The backup host 104 (e.g., first host), the sourcehost 106 (e.g., second host), and the client machine 111 may communicateover a network 112. The source host 106 (e.g., first host) may beembodied as a production system. The backup host 104 (e.g., second host)may execute a job that backs up a file set from the production system.The source host 106 (e.g., production system) includes a remote server114 that is communicatively coupled to a storage device 117 that storesa database 118 that, in turn, stores the file set 108. The backup host104 includes a local server 124 that is communicatively coupled to astorage device 126 that, in turn, includes a database 128 that stores acopy of the file set 108.

The backup host 104 may back up the source host 106 by executing a jobthat includes two steps. In the first step, the local server 124 fetchesmetadata for the file set 108 from the remote server 114. In the secondstep, the backup host 104 backs up the file set 108 based on themetadata. For example, part of the metadata may describe a directory. Inone embodiment, the part of the metadata may include a list of filenames describing files in the directory with their respective sizes andtypes. The backup host 104 may fetch the metadata from the source host106 by utilizing a thread from a thread pool. The backup host 104utilizes the thread to send a request to the remote server 114, receivea response from the remote server 114, and measure the latency of theround trip. In addition, the backup host 104 stores the latency andincrements a count of a number of requests in a sample in a movingaverage repository (not shown). For example, the backup host 104 maystore the latency and increment the count in a one-second sample. Themoving average depository may include multiple types of samples withdifferent sample periods. For example, the moving average depository mayinclude ten one-second samples, six ten-second samples, and sixtyone-minute samples. The backup host 104 further utilizes a timeout tocollect and aggregate the samples for the timeout period and othertimeout periods. For example, the backup host 104 responds to theexpiration of a one-second timeout by moving the current sampleidentifier to the next one second-sample and identifying whether thetenth one-second sample has been collected. If the tenth one-secondsample has been collected, then the backup host 104 aggregates the tenone-second samples together to form a ten-second sample and stores theten-second sample. The backup host 104 repeats these steps for the othersample periods.

The local server 124 may throttle the fetching of the metadata byincreasing and decreasing the size of the thread pool. Each thread isassociated with a process that is utilized by the local server 124 tosend a single request to the remote server 114, receive the responsefrom the remote server 114, and process the response. Accordingly, thelocal server 124 may increase the speed of the fetching by increasingthe size of the thread pool. In addition, the local server 124 maydecrease the speed of the fetching by decreasing the size of the threadpool. For example, a thread pool of eight threads facilitates theconcurrent sending, receiving, and processing of eight, parallel,retrievals of metadata items. The thread pool may be increased anddecreased responsive to the identification of predetermined conditions.For example, a predetermined condition may be defined based on anaverage latency of requests and an average number of requests for apredetermined period of time (e.g., one-minute sample), as describedlater in this document.

The system 100, may utilize adaptive throttling in a universal backuphost, according to an embodiment. For example, in requesting themetadata item, the local server 124 may utilize an operation from one ofmultiple operation sets. The local server 124 may identify the operationset based on the file sharing protocol that is shared by the source host106 and the backup host 104. For example, the local server 124 may senda request including an operation from a Unix operation set responsive toidentifying the local server 124 and the remote server 114 as utilizingthe Network File System protocol. In addition, the local server 124 maysend a request including an operation from a Windows operation setresponsive to identifying the local server 124 and the remote server 114as utilizing the Server Message Block protocol, and so forth.Accordingly, the backup host 104 exhibits itself to be “Universal”because it retrieves metadata and backs up file sets 108 from remoteservers 114 irrespective of their file sharing protocol.

A user may utilize the client machine 111 to request the backup host 104to back up the file set 108. For example, the user may utilize theclient machine 111 to schedule a periodic back-up of the file set 108from the source host 106 to the backup host 104. In addition, the usermay recover and restore the file set 108 from the backup host 104 to thesource host 106 at a point-in-time. For example, the client machine 111may receive the point-in-time identifying a date-time for restoring andrecovering the file set 108 from the backup host 104 to the source host106. In some embodiments, the file set 108 may be stored in replicate onthe source host 106 by utilizing remote server 109 and remote server110. Similarly, the file set 108 may be backed up in replicate on thebackup host 104 by utilizing the local server 124 and local server 132.In some embodiments, the remote server 114 may include multiple servers,the remote server 109 may include multiple servers, the remote server110 may include multiple servers, the local server 124 may includemultiple servers, the local server 130 may include multiple servers, andthe local server 132 may include multiple servers.

The database 118 may be a standalone database. For example, the database118 may not have replicates. In another embodiment, the database 118 maybe a cluster database embodied in replicates. For example, the sourcehost 106 may further include a remote server 109 and a remote server 110that are respectively communicatively coupled to storage devicesincluding replicate databases 118 that store replicate file sets 108.Similarly, the backup host 104 may include a local server 130 and alocal server 132 that are respectively communicatively coupled tostorage devices including replicate databases 128 that store replicatefile sets 108.

The networked system 102 may be embodied as a networked computingenvironment where the backup host 104, the source host 106, and theclient machine 111 are interconnected through one or more public and/orproprietary networks (e.g., Microsoft® provider of Azure Cloud ComputingPlatform & Services, Amazon provider of Amazon Web Services, and thelike). According to another embodiment, the system 100 may beimplemented as a single software platform. The backup host 104 may beembodied as a networked computing environment offered by Rubrik Inc., ofPalo Alto, Calif. For example, the backup host 104 may be implemented asa software platform that delivers backup, instant recovery, archival,search, analytics, compliance, and copy data management in one securefabric across data centers and clouds as offered by Rubrik Inc., of PaloAlto, Calif.

In some embodiments, the backup host 104 may back up a file set 108 thatis stored across multiple storage devices 117 respectively coupled tothe multiple remote servers 114 (e.g., PRIMARY). Similarly, in someembodiments, the backup host 104 may back up a file set 108 acrossmultiple storage devices 126 respectively coupled to multiple localservers 124 (e.g., PRIMARY). In one embodiment, the multiple remoteservers 114 (e.g., PRIMARY) and the multiple local servers 124 (e.g.,PRIMARY) may be embodied as network-attached storage (e.g., NAS). Inanother embodiment, the multiple remote servers 114 (e.g., PRIMARY) andthe multiple local servers 124 (e.g., PRIMARY) may be embodied asstand-alone hosts. In another embodiment, the multiple remote servers114 (e.g., PRIMARY) and the multiple local servers 124 (e.g., PRIMARY)may be embodied in different combinations of network-attached storage(e.g., NAS) and stand-alone hosts.

FIG. 1B is a block diagram illustrating a system 140, according to anembodiment, to utilize adaptive throttling. The system 140 may beutilized to process a file set 108 that is distributed over multipleremote servers 114. The system 140 corresponds to the system 100 in FIG.1A; accordingly, the same or similar references have been used toindicate the same or similar features unless otherwise indicated. Thesystem 140 includes a file set 108 split into four shards anddistributed across four remote servers 114 (e.g., PRIMARY). For example,the file set 108 may be a catalog organized alphabetically with a filecorresponding to each letter of the alphabet where a first remote server114 stores files “A” through “F;” a second remote server 114 storesfiles “G” through “M;” the third remote server 114 stores files “N”through “U;” and the fourth remote server 114 stores files “V” throughZ.”

FIG. 1C is a block diagram illustrating a system 150, according to anembodiment, to utilize adaptive throttling. The system 150 may beutilized to communicate with remote servers 114 utilizing different filesharing protocols. The system 150 corresponds to the system 100 in FIG.1A; accordingly, the same or similar references have been used toindicate the same or similar features unless otherwise indicated. Thelocal server 124 includes a job module 152 (e.g., job), a processingmodule 154, a thread pool 156, and a moving average repository 158. Thejob module 152 may be utilized to back up the file set 108 from the oneor more remote servers 114. The job may be triggered periodically by ascheduler or incidentally by a user who enters a command or makes aselection from a client machine, as previously described. The processingmodule 154 may include an agent server, a moving average manager, and athread pool manager.

The agent server utilizes the thread pool 156 to concurrently sendrequests 160, in parallel, to the remote servers 114, receive responses162 to the requests 160, and process the responses 162. For example, theagent server may utilize a thread pool 156 including ten threads toconcurrently send ten requests 160, in parallel, to the remote servers114. Accordingly, the size of the thread pool 156 determines the maximumlimit for the concurrent sending of requests 160, in parallel, to theremote servers 114. The agent server utilizes one thread to send arequest 160, receive a response 162 to the request 160, and process theresponse. The agent server stores an operation 164 in the request 160that, in turn, is executed on the remote server 114 to generate ametadata item 166 (e.g., portion of data) that, in turn, is included ina response 162. The agent server may select the operation from one ofmultiple operation sets. The agent server may identify the operation setbased on the file sharing protocol being utilized by the remote server114. For example, the agent server may send a request including anoperation 164 from a Unix operation set responsive to identifying theremote server 114 as utilizing the Network File System file sharingprotocol. In addition, the agent server may send a request including anoperation 164 from a Windows operation set responsive to identifying theremote server 114 as utilizing the Server Message Block protocol, and soforth. Accordingly, the backup host 104 exhibits itself as “Universal”because it includes a local server 124 that may retrieve a metadata item166 from remote servers 114 irrespective of the file sharing protocolthat is being utilized by the remote servers 114. The agent serverreceives the response 162, stores the metadata item 166 in the movingaverage repository 158, and increments a number of requests in themoving average repository 158. For example, the agent server may receivethe response 162, stores the metadata item 166 in a sample in a movingaverage (e.g., ten second samples) in the moving average repository 158,increments a number of requests in the sample.

The remote server 114 includes a backup agent 168 that receives therequest 160, executes the operation 164 to generate the metadata item166, stores the metadata item 166 in the response 162, and communicatesthe response 162 to the local server 124. For example, the backup agent168 may utilize the Unix operation set and the Network File System filesharing protocol. In another example, the backup agent 168 utilizes aWindows operation set and a Server Message Block file sharing protocol,and so forth.

The moving average manager aggregates the samples in the moving averagesresponsive to a timeout. For example, the moving average managerresponds to the expiration of a one-second timeout by moving a currentsample identifier to the next sample and identifying whether the tenthone-second sample has been collected. If the tenth one-second sample hasbeen collected, then the moving average manager aggregates the tenone-second samples together to form one ten-second sample and stores theten-second sample in the appropriate moving average (e.g., ten-secondsamples). In addition, if six ten-second samples have been collected,then the moving average manager aggregates the six ten-second samplestogether to form one one-minute sample and stores the one-minute samplein a moving average (e.g., one minute samples), and so forth.

The thread pool manager resizes the thread pool based on theaggregating. For example, the thread pool manager may resize the threadpool responsive to aggregating a ten-second sample and/or aggregating aone-minute sample. The thread pool manager may resize the thread pool byincreasing the number threads in the thread pool or by decreasing thenumber of threads in the thread pool, as described later in thisdocument.

FIG. 1D is a block diagram illustrating a moving average 170, accordingto an embodiment. The moving average 170 stores initial samples 172(e.g., one-second samples). Each of the samples are for a one-secondduration. The moving average 170 includes ten initial samples 172 (e.g.,samples). The samples are respectively identified “TIME-0′,” “TIME-1′,”“TIME-2′,” “TIME-3′,” “TIME-4′,” “TIME-5′” “TIME-6′,” “TIME-7′,”“TIME-8′,” and “TIME-9′.” The current sample identifier 174 identifies asample that is currently being used to storing latencies and count thenumber of requests. For example, responsive to receiving a response 162from a remote server 114, the agent server stores a latency 176 andincrements the number of requests 178 in the sample identified by thecurrent sample identifier 174 (e.g., “TIME-9”). The moving average 171is a circular buffer.

FIG. 1E is a block diagram illustrating a moving average 180, accordingto an embodiment, for storing one-second samples. The moving average 180corresponds to the moving average 170 in FIG. 1D accordingly, the sameor similar references have been used to indicate the same or similarfeatures unless otherwise indicated. Responsive to receiving a timeout(e.g., for a one-second period), the moving average manager advances thecurrent sample identifier 174 by one and identifies whether the movingaverage 180 is full. For example, if the moving average manageridentifies the moving average 180 is full (e.g., current sampleidentifier 174 equal to “TIME-0”), then the moving average manageraggregates (adds) the latencies 176 in the moving average 180 togenerate a sum and divides the sum by the number of samples (e.g., ten)to generate an average latency request 182 (e.g., average latency ofrequests) for the moving average 180. In addition, the moving averagemanager aggregates (adds) the number of requests 178 in the movingaverage 180 to generate a sum and divides the sum by the number ofsamples (e.g., ten) to generate an average number of requests 184 forthe moving average 180. Finally, the moving average manager stores theaverage latency requests 182 and the average number of requests 184 in amoving average that stores the appropriate size samples (e.g.,ten-second samples) and sends an event to the thread pool managersignaling the generation of the sample (e.g., ten-second sample), asdescribed and illustrated in FIG. 1F.

FIG. 1F is a block diagram illustrating a moving average 188, accordingto an embodiment. The moving average 188 is a circular buffer thatincludes six subsequent samples 186 (e.g., samples). Each of the samplesare for a ten-second duration. Each of the samples include the averagelatency request 182 and the average number of requests 184. The samplesare respectively identified “TIME-0′,” “TIME-10′,” “TIME-20′,”“TIME-30′,” “TIME-40′,” and “TIME-50′.” The moving average manageradvances to the next sample (e.g., “TIME-50”) with the current sampleidentifier 174, stores a sample in the moving average 188 at thelocation identified by the current sample identifier 174, and identifieswhether the moving average 188 is full (e.g., current sample identifier174 identifies the sample “TIME-0′”). For example, if the moving averagemanager identifies the moving average 188 is full, then the movingaverage manager generates a sample (e.g., one-minute). The movingaverage manager generates the sample (e.g., one-minute) by aggregating(adding) the average latency of requests 182 in the moving average 188in the subsequent samples 186 (e.g., six ten-second samples) to generatea sum and divides the sum by the number of samples (e.g., six) togenerate an average latency request 182. In addition, the moving averagemanager aggregates the average number of requests 184 in the subsequentsamples 186 to generate a sum and divides the sum by the number ofsamples (e.g., six) to generate an average number of requests 184.Finally, the moving average manager stores the sample (e.g., one-minute)that was aggregated in the appropriate moving average (e.g., one-minutesamples) and sends an event to the thread pool manager (e.g., one minutesample).

FIG. 1G is a block diagram illustrating a condition 190, according to anembodiment, to increase a size of the thread pool 156. If the condition190 is identified “TRUE” then the number of threads in the thread pool156 is increased. For example, if the condition 190 is identified “TRUE”then maximum threads 212 may be increased. The condition 190 includesthe average latency request 182 (ALR) and the average number of requests184 (ANOR) for the two most recent samples in a moving average. Thesubscript “1” identifies the most recent sample and the subscript “2”identifies the sample that was collected before the most recent sample.For example, FIG. 1F illustrates the moving average 188 as including thetwo most recent samples of the average latency of requests 182 with thesubscripts “1” and “2” and as including the two most recent samples ofthe average number of requests 184 with the subscripts “1” and “2.”Returning to FIG. 1G, the condition 190 includes three expressionsincluding an average latency ratio 191, an average number of requestsratio 193, and an increasing workload 195. If the thread pool manageridentifies the condition as being TRUE, then thread pool managerincreases the size of the thread pool 156. In one embodiment, the numberof threads may be increased by adding to maximum threads 212. Forexample, the thread pool manager may add one to maximum threads 212responsive to identifying the condition as being TRUE. In someembodiments, the number of threads added to maximum threads 212 isconfigurable. In one embodiment, the number of threads added to themaximum threads 212 may be increased by multiplying the number ofthreads in the thread pool 156 (e.g., current threads 214) by apercentage (e.g., 25%) to identify a number of threads to add to maximumthreads 212. In some embodiments, the percentage may be configurable.

FIG. 1H is a block diagram illustrating a condition 192, according to anembodiment, to decrease a size of a thread pool. If the condition 192 isidentified “TRUE” then the number of threads in the thread pool 156 isdecreased. For example, if the condition 192 is identified “TRUE” thenthe maximum threads 212 may be decreased. The condition 192 includes theaverage latency request 182 (ALR) and the average number of requests 184(ANOR) for the two most recent samples in a moving average. Thesubscript “1” identifies the most recent sample and the subscript “2”identifies the sample collected before the most recent sample, aspreviously described. The condition 190 includes two expressionsincluding the average latency ratio 191 and the average number ofrequests ratio 193. If the thread pool manager identifies the condition192 as being TRUE, then thread pool manager decreases the size of thethread pool 156. According to an embodiment, the number of threads inthe thread pool may be decreased by subtracting. For example, the threadpool manager may subtract one thread from the maximum threads 212responsive to identifying the condition 192 as being TRUE. In someembodiments, the number of threads subtracted from maximum threads 212is configurable. In one embodiment, the number of threads subtractedfrom maximum threads 212 may be decreased by multiplying the number ofthreads in the thread pool 156 (e.g., current threads 214) by apercentage (e.g., 25%) to identify the number of threads to subtractfrom the maximum threads 212. In some embodiments, the percentage may beconfigurable.

FIG. 2A is a block diagram illustrating a thread pool 156, according toan embodiment. The thread pool 156 includes one or more threads 200.Each thread 200 may be utilized to send a request 160, receive acorresponding response 162, and process the response 162. The size ofthe thread pool 156 may be increased or decreased to throttle theretrieving of the metadata item 166 from the source host 106, aspreviously described.

FIG. 2B is a block diagram illustrating a thread 200, according to anembodiment. The thread 200 includes a thread identifier 202, a remoteserver identifier 204, a transmit time 206, and a receive time 208. Thethread identifier 202 uniquely identifies the thread 200. The remoteserver identifier 204 identifies the remote server 114 associated with arequest 160. For example, the remote server identifier 204 may store auniversal resource locator (e.g., URL) identifying a location on thenetwork 112 of the remote server 114 that was sent the request 160. Thetransmit time 206 registers a time the request 160 was sent to theremote server 114. For example, the agent server may register a timestamp for the current time in the transmit time 206 responsive tosending a request 160 to the remote server 114. The receive time 208registers the time a response to the request is received from the remoteserver 114. For example, the agent server may register a time stamp forthe current time in the receive time 208 responsive to receiving theresponse from the remote server 114.

FIG. 2C is a block diagram illustrating a thread pool information 210,according to an embodiment. The thread pool information 210 includesmaximum threads 212 and current threads 214. The maximum threads 212 isthe maximum number of threads 200 that the job module 152 utilizes toconcurrently retrieve the metadata items 166, in parallel, from thesource host 106. The maximum threads 212 may be incremented by thethread pool manager based on the condition 190 for increasing the sizeof the thread pool 156 and decremented by the thread pool manager basedon the condition 192 for decreasing the size of the thread pool 156, aspreviously described. The current threads 214 is the current number ofthreads being utilized by the job module 152 to concurrently retrievethe metadata item 166, in parallel, from the source host 106. Forexample, a current thread 214 value of four indicates the job module 152has spawned four threads 200 (e.g., processes) of the agent server tocommunicate the requests 160, receive the responses 162 and process theresponses 162. The current threads may be incremented by the agentserver responsive to the agent server identifying the current threads214 as being less than the maximum threads 212. The current threads 214may be decremented by the agent server and responsive to the agentserver identifying the current threads 214 as being greater than themaximum threads 212.

FIG. 2D is a block diagram illustrating a moving average repository 158,according to an embodiment. The moving average repository 158 stores oneor more moving averages 171 that, in turn, store samples. For example,each of the moving averages 171 may respectively store samples that arecollected for a different period of time (e.g., one-second, ten-seconds,one-minute, ten-minutes, and so forth). The moving average 171associated with the shortest period of time stores initial samples 172and the remaining moving averages 171 store subsequent samples 186.

FIG. 2E is a block diagram illustrating moving average 171, according toan embodiment. The moving average 171 includes one or more initialsamples 172 (e.g., samples) and a current sample identifier 174. Thepresent example includes four initial samples 172 (e.g., TIME=“0,”TIME=“1,” TIME=“2,” TIME=“3”). The moving average 171 is a circularbuffer. Accordingly, the sample after “TIME=3” is the sample for“TIME=0.”

FIG. 2F is a block diagram illustrating the initial sample 172,according to an embodiment. The initial sample 172 includes latencyinformation 213 and number of requests 178. The latency information 213includes one or more latencies 176. Each latency 176 is a measure oftime that elapses from the sending of a request 160 to the receiving ofa corresponding response 162. The number of requests 178 is a count ofthe number of requests 160 sent to the remote server 114 for the initialsample 172 period (e.g., one second). The number of requests 178 and thenumber of latencies 176 should match.

FIG. 2G is a block diagram illustrating moving average 171, according toan embodiment. The moving average 171 includes one or more subsequentsamples 186 (e.g., samples) and a current sample identifier 174. Forexample, the moving average 171 in FIG. 2G includes four subsequentsamples 186 (e.g., TIME=“0,” TIME=“1,” TIME=“2,” TIME=“3”). The movingaverage manager may store and aggregate the subsequent samples 186. Forexample, the moving average manger may store a subsequent sample 186(e.g., sample) in the moving average 171 in accordance with the currentsample identifier 174 and identify whether the moving average 171 isfull. If the moving average manager identifies the moving average 171 isfull (e.g., TIME=“3”), then the moving average manager generates a newsubsequent sample 186 by aggregating the subsequent samples 186 in themoving average 171 and advancing the current sample identifier 174 tothe next initial sample (e.g., TIME=“0”).

FIG. 2H is a block diagram illustrating a subsequent sample 186,according to an embodiment. The subsequent sample 186 includes theaverage latency of requests 182 and the average number of requests 184.

FIG. 3A is a block diagram illustrating a method 330, according to anembodiment, for adaptive throttling. The method 330 for adaptivethrottling may be for a backup host 104 that is universal, according toan embodiment. The method 330 for adaptive throttling may be for asource host 106, according to an embodiment. The method 330 for adaptivethrottling may include the adaptive throttling communication protocol,according to an embodiment.

The method 330 commences, at the backup host 104, at the operation 332,with the job module 152 (e.g., job) responding to receipt of atriggering event. For example, the job may respond to receiving thetriggering event by executing to back up a file set 108 from the sourcehost 106 to the backup host 104. For example, the triggering event maybe sent periodically (e.g., noon each day, on the hour, each hour, andso forth) from a scheduler to back up the file set 108 “XYZ.” Also, forexample, a triggering event may be caused by a user who enters a commandfrom the client machine 111 (e.g., backup “XYZ”) or selects a userinterface element. Other embodiments include other triggering events. Insome embodiments, multiple job modules 152 may execute at the same timewith execution being utilized to back up a single file set 108.

At operation 334, the job module 152 sets a timeout to schedule anexecution of the moving average manager. For example, the job module 152may set a one-second timeout that causes a scheduler to wait one secondbefore sending a triggering event to the moving average manager causingthe moving average manager to execute.

At decision operation 336, the job module 152 identifies whether tospawn a thread of the agent server. If current threads 214 is less thanmaximum threads 212, the job module 152 spawns a thread of the agentserver and branches to operation 338. Otherwise, the job module 152branches to decision operation 342. At operation 338, the agent serverretrieves and processes the metadata item 166 as further described andillustrated in FIG. 3B. Returning to FIG. 3A, at operation 340, the jobmodule 152 may add a value of one or more to the current threads 214,according to an embodiment.

At decision operation 342, the job module 152 identifies whether theagent server is waiting for one or more responses 162. If the job module152 identifies the agent server is waiting for one or more responses162, processing continues at decision operation 342. Otherwise,processing continues at operation 352.

At operation 346, an expiration of a timeout (e.g., one second) causesthe moving average manager to execute. The moving average manager mayaggregate samples in one or more moving averages 171 in the movingaverage repository 158 responsive to the timeout. For example, themoving average manager may aggregate samples in a one-second movingaverage 171 to generate a ten-second sample, and/or aggregate samples ina ten-second moving average 171 to generate a one minute sample, and/oraggregate samples in a one-minute moving average 171 to generate a onehour sample, and so forth. The operation 346 is further illustrated inFIG. 5 and described in the written description.

At operation 348, the thread pool manager may resize the thread pool156. For example, the thread pool manager may resize the thread pool 156responsive to receive a triggering event from the moving average managerindicating the moving average manager generated a new sample (e.g.,one-second sample, ten-second sample, and so forth). At operation 350,the moving average manager resets the timeout (e.g., one second). Atoperation 352, the job module 152 backs up the file set 108 from thesource host 106 to the backup host 104 based on the metadata.

FIG. 3B is a block diagram illustrating a method 360, according to anembodiment, for communicating requests 160 in parallel, receivingresponses 162, and processing the responses 162. Illustrated on the leftare operations performed by the local server 124 and illustrated on theright are operations performed by the remote server 114. The method 360commences at operation 362, on the local server 124, with the agentserver, executing as one thread 200, identifying an operation set (e.g.,first operation set) from multiple operation sets. The agent server mayidentify the operation set based on a file sharing protocol. Forexample, the agent server may identify the operation set (e.g., Windows)responsive to the agent server identifying file sharing protocol (e.g.,Server Message Block) is being utilized by the remote server 114. Atoperation 364, the agent server communicates a request 160, over thenetwork 112, to the remote server 114. For example, the agent server maystore an operation 164 (e.g., “dir” Windows Command) in the request 160,store a URL for the remote server 114 in the remote server identifier204 in the thread 200, retrieve the current time (e.g., timestamp) fromthe operating system, store the timestamp in the transmit time 206 ofthe thread 200, and communicate the request 160 to the remote server114.

At operation 366, at the remote server 114, the backup agent 168receives the request 160. At operation 368, the backup agent 168executes the operation 164 in the request 160. For example, the backupagent 168 may extract the operation 164 from the request 160 and executethe operation 164 to generate the metadata item 166. At operation 370,the backup agent 168 stores the metadata item 166 in the response 162and communicates the response 162 to the local server 124.

At operation 372, at the local server 124, the agent server receives aresponse 162. For example, the agent server may receive the response162, retrieve the current time (e.g., timestamp) from the operationsystem, and store the timestamp in the receive time 208 in thecorresponding thread 200. At operation 374, the agent server processesthe response 162. For example, the agent server may process the response162 by computing (e.g., generating) and storing the latency 176 in theinitial sample 172 and incrementing the number of requests 178 in theinitial sample 172. The operation 374 is further illustrated indescribed FIG. 3C. At decision operation 376, the agent serveridentifies whether to reduce the number of threads 200. For example, ifthe agent server identifies the maximum threads 212 is less than thecurrent threads 214, then the agent server branches to operation 380.Otherwise, the agent server branches to operation 382. At the off-pageconnector “B” 378, the agent server receives control from the threadpool manager. For example, the agent server may receive a triggeringevent from the thread pool manager to reduce the number of threads 200.At operation 380, the agent server subtracts one or more threads fromthe current threads 214 based on a percentage. According to anembodiment, the agent server may multiply the current threads 214 bytwenty percent to identify a value that is subtracted from the currentthreads 214. For example, if the current threads 214 equals ten, thenthe agent server may multiply the current threads 214 (e.g., ten) bytwenty percent to identify the value, two, that is subtracted from thecurrent threads 214 (e.g., (ten*twenty percent is two) (e.g, (ten−two)is eight). At decision operation 382, the agent server identifieswhether there are more requests 160 for communicating to the remoteserver 114. If the agent server identifies there are more requests 160for communicating to the remote server 114, then processing branches tooperation 362. Otherwise, processing ends.

FIG. 3C is a block diagram illustrating a method 383, according to anembodiment, for processing the response 162. The method 383 commences atoperation 384, with the agent server generating a latency 176, asfurther illustrated and described in FIG. 4. At operation 386, the agentserver increments the number of requests 178 in the appropriate sample.For example, the agent server may identify an initial sample 172 (e.g.,sample) in the moving average 171 (e.g., one-second samples) based onthe current sample identifier 174 and increment the number of requests178 in the initial sample 172 by “1.”

FIG. 4 is a block diagram illustrating a method 388, according to anembodiment, for generating the latency 176. At operation 390, the agentserver computes the latency 176. For example, the agent server subtractstransmit time 206, in the thread 200, from the receive time 208, in thethread 200, to compute the latency 176. At operation 392, the agentserver stores the latency 176 in the initial sample 172 (sample). Forexample, the agent server identifies the sample with the current sampleidentifier 174 in the moving average 171 (e.g., one second samples) andstores the latency 176 in the initial sample 172.

FIG. 5 is a block diagram illustrating a method 400, according to anembodiment, for aggregating latencies 176 and number of requests 178. Atoperation 402, the moving average manager advances to the next initialsample 172 (e.g., sample). For example, the moving average manager mayincrement the current sample identifier 174 by “1.” At decisionoperation 406, the moving average manager identifies whether the movingaverage 171 is full. For example, the moving average manager mayidentify whether the current sample identifier 174 points to the initialsample 172 that is first in the moving average. If the moving averagemanager identifies the initial sample 172 is first (e.g., see FIG. 2E,initial sample 172 associated with TIME=“0”) in the moving average 171,then processing continues at operation 408. Otherwise processing ends.At operation 408, the moving average manager aggregates the samples. Forexample, the moving average manager adds each latency 176 in the movingaverage 171 to generate a sum and divides the sum by the number ofsamples in the moving average 171 to generate the average latency ofrequests 182. In addition, the moving average manager adds each of thenumber of requests 178 in the moving average 171 to generate a sum anddivides the sum by the number of samples in the moving average 171 togenerate the average number of requests 184. In the present example, themoving average 171 may include ten one-second samples. Accordingly, themoving average manager divides the sum of the number of latencies 176 byten to generate the average latency of requests 182 and divides the sumof the number of requests 178 by ten to generate the average number ofrequests 184. At operation 410, the moving average manager advances tothe next sample in the next moving average 171. For example, the movingaverage manager may increment the current sample identifier 174 in themoving average 171 (e.g., ten-second samples) by “1” and tests for awrap-around condition. If a wrap-around condition is identified, themoving average manager initializes the current sample identifier 174 tozero. At operation 412, the moving average manager stores the sample(e.g., ten-second sample) in the appropriate moving average 171 (e.g.,moving average 171 storing ten-second samples). At operation 414, themoving average manager communicates a triggering event to the threadpool manager. For example, the moving average manager may communicate atriggering event indicating a ten-second sample was generated (see FIG.6, at off-page connector “C”).

At decision operation 416, the moving average manager processes the nextmoving average 171 (e.g., ten-second samples) and begins by identifyingwhether the moving average 171 (e.g., ten-second samples) is full. Forexample, the moving average manager may identify whether the currentsample identifier 174 points to the subsequent sample 186 (e.g., sample)that is the first in the moving average 171. If the moving averagemanager identifies the first sample (e.g., see FIG. 2G, subsequentsample 186 associated with TIME=“0”) in the moving average 171, thenprocessing continues at operation 418. Otherwise processing ends. Atoperation 418, the moving average manager aggregates the samples. Forexample, the moving average manager adds each average latency ofrequests 182 in the moving average 171 to generate a sum of the latencyof requests 182 and divides the sum by the number of samples (e.g., six)to generate an average latency of requests 182. In addition, the movingaverage manager adds each of the average number of requests 184 in themoving average 171 to generate a sum of the average number of requests184 and divides the sum by the number of samples (e.g., six) to generatean average number of requests 184. In the present example, the movingaverage 171 may include six ten-second samples. Accordingly, the movingaverage manager divides the sum of the average latency of requests 182by six to generate the average latency of requests 182 and divides thesum of the average number of requests 184 by six to generate the averagenumber of requests 184. At operation 420, the moving average manageradvances to the next sample in the next moving average 171. For example,the moving average manager may increment the current sample identifier174 in the moving average 171 (e.g., one minute samples) by “1” andtests for a wrap-around condition. If a wrap-around condition isidentified, the moving average manager initializes the current sampleidentifier 174 to zero. At operation 422, the moving average managerstores the sample (e.g., one minute) in the appropriate moving average171 (e.g., moving average 171 storing one-minute samples). At operation424, the moving average manager communicates a triggering event to thethread pool manager. For example, the moving average manager maycommunicate a triggering event indicating that a one-minute sample wasgenerated (see FIG. 6, at off-page connector “D”). Other embodiments mayinclude additional moving averages 171 (e.g., ten-minute, one hour, andso forth).

FIG. 6 is a block diagram illustrating a method 430, according to anembodiment, to resize the thread pool. The method 430 commences, atoperation 432, at the local server 124, with the thread pool managerretrieving the two most recent samples of the average latency ofrequests 182. For example, as illustrated in FIG. 1F, the thread poolmanager may utilize current sample identifier 174 to retrieve theaverage latency of requests 182 (ALR₁) at the subsequent sample 186 forTIME-50″ and the average latency of requests 182 (ALR₂) at thesubsequent sample 186 for TIME-40″ from the moving average 171. Thethread pool manager executes the operation 432 responsive to receiving atriggering event. For example, at off-page connecter 434 “C,” the threadpool manager may receive a triggering event indicating a ten-secondsample was aggregated. Accordingly, the thread pool manager responds tothe triggering event, at operation 432, by utilizing the current sampleidentifier 174 associated with the moving average 171 that storesten-second samples. Also, for example, at off-page connecter 436 “D,”the thread pool manager may receive a triggering event indicating aone-minute sample was aggregated. Accordingly, the thread pool managerresponds to the triggering event for the one-minute sample, at operation432, by utilizing the current sample identifier 174 associated with themoving average 171 that stores one-minute samples. At operation 438, thethread pool manager computes the average latency ratio 191 based on theaverage latency of requests 182 (ALR₁) and average latency of requests182 (ALR₂) that were retrieved.

At operation 440, the thread pool manager retrieves the two most recentsamples of the average number of requests 184. For example, the threadpool manager may utilize current sample identifier 174 to retrieve theaverage number of requests 184 (ANOR₁), at the subsequent sample 186(e.g., sample) TIME-50,” and the average number of requests 184 (ANOR₂),at the subsequent sample 186 identified TIME-40,” from the movingaverage 171 illustrated in FIG. 1F. At operation 442, The thread poolmanager computes the average number of requests ratio 193 and theincreasing workload 195 based on the average number of requests 184(ANOR₁) and average number of requests 184 (ANOR₂) that were retrieved.

At decision operation 444, the thread pool manager identifies whetherthe condition for increasing the size of the thread pool 156 is TRUE. Ifthe condition for increasing the size of the thread pool 156 is TRUE,then processing continues at operation 446. Otherwise processingcontinues at decision operation 448. At operation 446, the thread poolmanager increases maximum threads 212, as previously described inassociation with FIG. 1G, and sends an event to the agent server thatindicates the maximum size of the thread pool 156 has been increased(see FIG. 3A, at off-page connector “A”).

At decision operation 448, the thread pool manager identifies whetherthe condition for decreasing the size of the thread pool 156 is TRUE. Ifthe condition for decreasing the size of the thread pool is TRUE, thenprocessing continues at operation 450. Otherwise processing ends. Atoperation 450, the thread pool manager decreases maximum threads 212, aspreviously described in association with FIG. 1F, and sends an event tothe agent server that indicates the maximum size of the thread pool 156has been decreased (see FIG. 3B, off-page connector

Adaptive Throttling for a Source Host

According to an embodiment, systems, methods, and machine storagemediums for adaptive throttling for a source host are described. Anexample system may comprise: executing a job, at a backup host, to backup a file set from a source host responsive to a triggering event, thebacking up of the file set including fetching metadata from the sourcehost; communicating, in parallel, one or more requests, over a network,to the source host by utilizing one or more threads from a thread pool,the communicating the one or more requests including communicating afirst request, over the network, to the source host by utilizing a firstthread to fetch a first metadata item, the first request being based ona file sharing protocol and an operation set, the first requestincluding a first operation; receiving responses, over the network,corresponding to the one or more requests by utilizing the one or morethreads, the receiving the responses including receiving a firstresponse corresponding to the first request by utilizing the firstthread, the first response including the first metadata item; processingthe responses including processing the first response, comprising:generating a first latency based on the first response, and incrementinga number of requests based on the first response; aggregating samplesresponsive to a timeout, the samples including the first latency and thenumber of requests; resizing the thread pool based on the aggregating;and backing up the file set from the source host based on the metadata.

The example system described above, wherein the file sharing protocol isNetwork File System protocol and the operation set is a Unix operationset. The example system described above, wherein the file sharingprotocol is Server Message Block protocol and the operation set is aWindows operation set. The example system described above, wherein thefirst request includes the first operation, for execution on the sourcehost, to generate the first metadata item. The example system describedabove, wherein the generating the first latency further comprises:identifying a receive time responsive to the first thread receiving thefirst response; subtracting a transmit time from the receive time togenerate the first latency; and storing the first latency in a firstsample in a first moving average including a first plurality of samples.The example system described above, wherein the aggregating the samplesfurther comprises: aggregating the first latency over the firstplurality of samples to generate a first average latency of requests;storing the first average latency of requests in a first sample in asecond moving average including a second plurality of samples;aggregating the number of requests over the first plurality of samplesto generate a first average number of requests; and storing the firstaverage number of requests in a first sample in the second movingaverage including the second plurality of samples. The example systemdescribed above, wherein the resizing further comprises computing anaverage latency ratio based on the first average latency of requests andthe second average latency of requests; and computing a number ofrequests ratio based on the first average number of requests and thesecond average number of requests. The example system described above,wherein the resizing further comprises increasing a size of the threadpool responsive to identifying the average number of requests ratio asbeing greater than the average latency ratio and wherein the increasingthe size of the thread pool includes increasing the size of the threadpool by one thread. The example system described above, wherein theresizing further comprises decreasing a size of the thread poolresponsive to identifying the average number of requests ratio as beingless than the average latency ratio and wherein the decreasing the sizeof the thread pool includes decreasing the thread pool by a percentageof the size of the thread pool and wherein the percentage isconfigurable.

An example method may comprise: executing a job, at a backup host, toback up a file set from a source host responsive to a triggering event,the backing up of the file set including fetching metadata from thesource host; communicating, in parallel, one or more requests, over anetwork, to the source host by utilizing one or more threads from athread pool, the communicating the one or more requests includingcommunicating a first request, over the network, to the source host byutilizing a first thread to fetch a first metadata item, the firstrequest being based on a file sharing protocol and an operation set, thefirst request including a first operation; receiving responses, over thenetwork, corresponding to the one or more requests by utilizing the oneor more threads, the receiving the responses including receiving a firstresponse corresponding to the first request by utilizing the firstthread, the first response including the first metadata item; processingthe responses including processing the first response, comprising:generating a first latency based on the first response, and incrementinga number of requests based on the first response; aggregating samplesresponsive to a timeout, the samples including the first latency and thenumber of requests; resizing the thread pool based on the aggregating;and backing up the file set from the source host based on the metadata.The example method above, wherein the file sharing protocol is NetworkFile System protocol and the operation set is a Unix operation set. Theexample method above, wherein the file sharing protocol is ServerMessage Block protocol and the operation set is a Windows operation set.The example method above, wherein the first request includes the firstoperation, for execution on the source host, to generate the firstmetadata item. The example method above, wherein the generating thefirst latency further comprises: identifying a receive time responsiveto the first thread receiving the first response; subtracting a transmittime from the receive time to generate the first latency; and storingthe first latency in a first sample in a first moving average includinga first plurality of samples. The example method above, wherein theaggregating the samples further comprises: aggregating the first latencyover the first plurality of samples to generate a first average latencyof requests; storing the first average latency of requests in a firstsample in a second moving average including a second plurality ofsamples; aggregating the number of requests over the first plurality ofsamples to generate a first average number of requests; and storing thefirst average number of requests in a first sample in the second movingaverage including the second plurality of samples. The example methodabove, wherein the resizing further comprises computing an averagelatency ratio based on the first average latency of requests and thesecond average latency of requests; and computing a number of requestsratio based on the first average number of requests and the secondaverage number of requests. The example method above, wherein theresizing further comprises increasing a size of the thread poolresponsive to identifying the average number of requests ratio as beinggreater than the average latency ratio and wherein the increasing thesize of the thread pool includes increasing the size of the thread poolby one thread. The example method above, wherein the resizing furthercomprises decreasing a size of the thread pool responsive to identifyingthe average number of requests ratio as being less than the averagelatency ratio and wherein the decreasing the size of the thread poolincludes decreasing the thread pool by a percentage of the size of thethread pool and wherein the percentage is configurable.

An example machine-storage medium may comprise the machine-storagemedium and storing a set of instructions that, when executed by aprocessor, causes a machine to perform operations comprising: executinga job, at a backup host, to back up a file set from a source hostresponsive to a triggering event, the backing up of the file setincluding fetching metadata from the source host; communicating, inparallel, one or more requests, over a network, to the source host byutilizing one or more threads from a thread pool, the communicating theone or more requests including communicating a first request, over thenetwork, to the source host by utilizing a first thread to fetch a firstmetadata item, the first request being based on a file sharing protocoland an operation set, the first request including a first operation;receiving responses, over the network, corresponding to the one or morerequests by utilizing the one or more threads, the receiving theresponses including receiving a first response corresponding to thefirst request by utilizing the first thread, the first responseincluding the first metadata item; processing the responses includingprocessing the first response, comprising: generating a first latencybased on the first response, and incrementing a number of requests basedon the first response; aggregating samples responsive to a timeout, thesamples including the first latency and the number of requests; resizingthe thread pool based on the aggregating; and backing up the file setfrom the source host based on the metadata.

The example machine-storage medium above, wherein the file sharingprotocol is Network File System protocol and the operation set is a Unixoperation set.

Adaptive Throttling Communication Protocol

According to an embodiment, systems, methods, and machine storagemediums for an adaptive throttling communication protocol are described.An example system may comprise: at least one processor and memory havinginstructions that, when executed, cause the at least one processor toperform operations comprising: communicating, in parallel, one or morerequests, over a network, from a first host to a second host byutilizing one or more threads from a thread pool, the communicating theone or more requests including communicating a first request, over thenetwork, to the second host by utilizing a first thread to fetch a firstportion of data, the first request being based on a file sharingprotocol and an operation set, the first request including a firstoperation; receiving responses, over the network, corresponding to theone or more requests by utilizing the one or more threads, the receivingthe responses including receiving a first response corresponding to thefirst request by utilizing the first thread, the first responseincluding the first portion of data; processing the responses includingprocessing the first response, comprising: generating a first latencybased on the first response, and incrementing a number of requests basedon the first response; aggregating samples responsive to a timeout, thesamples including the first latency and the number of requests; andresizing the thread pool based on the aggregating.

The example system above, wherein the file sharing protocol is NetworkFile System protocol and the operation set is a Unix operation set. Theexample system above, wherein the file sharing protocol is ServerMessage Block protocol and the operation set is a Windows operation set.The example system above, wherein the first request includes the firstoperation, for execution on the second host, to generate a portion ofthe data. The example system above, wherein the generating the firstlatency further comprises: identifying a receive time responsive to thefirst thread receiving the first response; subtracting a transmit timefrom the receive time to generate the first latency; and storing thefirst latency in a first sample in a first moving average including afirst plurality of samples. The example system above, wherein theaggregating the samples further comprises: aggregating the first latencyover the first plurality of samples to generate a first average latencyof requests; storing the first average latency of requests in a firstsample in a second moving average including a second plurality ofsamples; aggregating the number of requests over the first plurality ofsamples to generate a first average number of requests; and storing thefirst average number of requests in a first sample in the second movingaverage including the second plurality of samples. The example systemabove, wherein the resizing further comprises computing an averagelatency ratio based on the first average latency of requests and thesecond average latency of requests; and computing a number of requestsratio based on the first average number of requests and the secondaverage number of requests. The example system above, wherein theresizing further comprises increasing a size of the thread poolresponsive to identifying the average number of requests ratio as beinggreater than the average latency ratio and wherein the increasing thesize of the thread pool includes increasing the size of the thread poolby one thread. The example system above, wherein the resizing furthercomprises decreasing a size of the thread pool responsive to identifyingthe average number of requests ratio as being less than the averagelatency ratio and wherein the decreasing the size of the thread poolincludes decreasing the thread pool by a percentage of the size of thethread pool and wherein the percentage is configurable.

An example method comprising: communicating, in parallel, one or morerequests, over a network, from a first host to a second host byutilizing one or more threads from a thread pool, the communicating theone or more requests including communicating a first request, over thenetwork, to the second host by utilizing a first thread to fetch a firstportion of data, the first request being based on a file sharingprotocol and an operation set, the first request including a firstoperation; receiving responses, over the network, corresponding to theone or more requests by utilizing the one or more threads, the receivingthe responses including receiving a first response corresponding to thefirst request by utilizing the first thread, the first responseincluding the first portion of data; processing the responses includingprocessing the first response, comprising: generating a first latencybased on the first response, and incrementing a number of requests basedon the first response; aggregating samples responsive to a timeout, thesamples including the first latency and the number of requests; andresizing the thread pool based on the aggregating.

The example method above, wherein the file sharing protocol is NetworkFile System protocol and the operation set is a Unix operation set. Theexample method above, wherein the file sharing protocol is ServerMessage Block protocol and the operation set is a Windows operation set.The example method above, wherein the first request includes the firstoperation, for execution on the second host, to generate a portion ofthe data. The example method above, wherein the generating the firstlatency further comprises: identifying a receive time responsive to thefirst thread receiving the first response; subtracting a transmit timefrom the receive time to generate the first latency; and storing thefirst latency in a first sample in a first moving average including afirst plurality of samples. The example method above, wherein theaggregating the samples further comprises: aggregating the first latencyover the first plurality of samples to generate a first average latencyof requests; storing the first average latency of requests in a firstsample in a second moving average including a second plurality ofsamples; aggregating the number of requests over the first plurality ofsamples to generate a first average number of requests; and storing thefirst average number of requests in a first sample in the second movingaverage including the second plurality of samples. The example methodabove, wherein the resizing further comprises computing an averagelatency ratio based on the first average latency of requests and thesecond average latency of requests; and computing a number of requestsratio based on the first average number of requests and the secondaverage number of requests. The example method above, wherein theresizing further comprises increasing a size of the thread poolresponsive to identifying the average number of requests ratio as beinggreater than the average latency ratio and wherein the increasing thesize of the thread pool includes increasing the size of the thread poolby one thread. The example method above, wherein the resizing furthercomprises decreasing a size of the thread pool responsive to identifyingthe average number of requests ratio as being less than the averagelatency ratio and wherein the decreasing the size of the thread poolincludes decreasing the thread pool by a percentage of the size of thethread pool and wherein the percentage is configurable.

An example machine-storage medium for storing a set of instructionsthat, when executed by a processor, causes a machine to performoperations comprising: communicating, in parallel, one or more requests,over a network, from a first host to a second host by utilizing one ormore threads from a thread pool, the communicating the one or morerequests including communicating a first request, over the network, tothe second host by utilizing a first thread to fetch a first portion ofdata, the first request being based on a file sharing protocol and anoperation set, the first request including a first operation; receivingresponses, over the network, corresponding to the one or more requestsby utilizing the one or more threads, the receiving the responsesincluding receiving a first response corresponding to the first requestby utilizing the first thread, the first response including the firstportion of data; processing the responses including processing the firstresponse, comprising: generating a first latency based on the firstresponse, and incrementing a number of requests based on the firstresponse; aggregating samples responsive to a timeout, the samplesincluding the first latency and the number of requests; and resizing thethread pool based on the aggregating.

The example machine-storage medium above, wherein the file sharingprotocol is Network File System protocol and the operation set is a Unixoperation set.

FIG. 7A depicts one embodiment of a networked computing environment 1100in which the disclosed technology may be practiced. As depicted, thenetworked computing environment 1100 includes a datacenter 1150, astorage appliance 1140, and a computing device 1154 in communicationwith each other via one or more networks 1180. The networked computingenvironment 1100 may include a plurality of computing devicesinterconnected through one or more networks 1180. The one or morenetworks 1180 may allow computing devices and/or storage devices toconnect to and communicate with other computing devices and/or otherstorage devices. In some cases, the networked computing environment 1100may include other computing devices and/or other storage devices notshown. The other computing devices may include, for example, a mobilecomputing device, a non-mobile computing device, a server, awork-station, a laptop computer, a tablet computer, a desktop computer,or an information processing system. The other storage devices mayinclude, for example, a storage area network storage device, anetworked-attached storage device, a hard disk drive, a solid-statedrive, or a data storage system.

The datacenter 1150 may include one or more servers, such as server1160, in communication with one or more storage devices, such as storagedevice 1156. The one or more servers 1160 may also be in communicationwith one or more storage appliances, such as storage appliance 1170. Theserver 1160, storage device 1156, and storage appliance 1170 may be incommunication with each other via a networking fabric connecting serversand data storage units within the datacenter 1150 to each other. Thestorage appliance 1170 may include a data management system for backingup virtual machines and/or files within a virtualized infrastructure.The server 1160 may be used to create and manage one or more virtualmachines associated with a virtualized infrastructure.

The one or more virtual machines may run various applications, such as adatabase application or a web server. The storage device 1156 mayinclude one or more hardware storage devices for storing data, such as ahard disk drive (HDD), a magnetic tape drive, a solid-state drive (SSD),a storage area network (SAN) storage device, or a networked attachedstorage (NAS) device. In some cases, a data center, such as datacenter1150, may include thousands of servers and/or data storage devices incommunication with each other. The data storage devices may comprise atiered data storage infrastructure (or a portion of a tiered datastorage infrastructure). The tiered data storage infrastructure mayallow for the movement of data across different tiers of a data storageinfrastructure between higher-cost, higher-performance storage devices(e.g., solid-state drives and hard disk drives) and relativelylower-cost, lower-performance storage devices (e.g., magnetic tapedrives).

The one or more networks 1180 may include a secure network such as anenterprise private network, an unsecure network such as a wireless opennetwork, a local area network (LAN), a wide area network (WAN), and theInternet. The one or more networks 1180 may include a cellular network,a mobile network, a wireless network, or a wired network. Each networkof the one or more networks 1180 may include hubs, bridges, routers,switches, and wired transmission media such as a direct-wiredconnection. The one or more networks 1180 may include an extranet orother private network for securely sharing information or providingcontrolled access to applications or files.

A server, such as server 1160, may allow a client to downloadinformation or files (e.g., executable, text, application, audio, image,or video files) from the server or to perform a search query related toparticular information stored on the server. In some cases, a server mayact as an application server or a file server. In general, a server mayrefer to a hardware device that acts as the host in a client-serverrelationship or a software process that shares a resource with orperforms work for one or more clients.

One embodiment of server 1160 includes a network interface 1165,processor 1166, memory 1167, disk 1168, a virtualization manager 1169,and a deduplication system 1171 (e.g., lightweight deduplication system)all in communication with each other. Network interface 1165 allowsserver 1160 to connect to one or more networks 1180. Network interface1165 may include a wireless network interface and/or a wired networkinterface. Processor 1166 allows server 1160 to executecomputer-readable instructions stored in memory 1167 in order to performprocesses described herein. Processor 1166 may include one or moreprocessing units, such as one or more CPUs and/or one or more GPUs.Memory 1167 may comprise one or more types of memory (e.g., RAM, SRAM,DRAM, ROM, EEPROM, Flash, etc.). Disk 1168 may include a hard disk driveand/or a solid-state drive. Memory 1167 and disk 1168 may comprisehardware storage devices.

The virtualization manager 1169 may manage a virtualized infrastructureand perform management operations associated with the virtualizedinfrastructure. The virtualization manager 1169 may manage theprovisioning of virtual machines running within the virtualizedinfrastructure and provide an interface to computing devices interactingwith the virtualized infrastructure. In one example, the virtualizationmanager 1169 may set a virtual machine into a frozen state in responseto a snapshot request made via an application programming interface(API) by a storage appliance, such as storage appliance 1170. Settingthe virtual machine into a frozen state may allow a point-in-timesnapshot of the virtual machine to be stored or transferred. In oneexample, updates made to a virtual machine that has been set into afrozen state may be written to a separate file (e.g., an update file)while the virtual machine may be set into a read-only state to preventmodifications to the virtual disk file while the virtual machine is inthe frozen state.

The virtualization manager 1169 may then transfer data associated withthe virtual machine (e.g., an image of the virtual machine or a portionof the image of the virtual disk file associated with the state of thevirtual disk at the point-in-time from which it is frozen) to a storageappliance in response to a request made by the storage appliance. Afterthe data associated with the point-in-time snapshot of the virtualmachine has been transferred to the storage appliance 1170, the virtualmachine may be released from the frozen state (i.e., unfrozen) and theupdates made to the virtual machine and stored in the separate file maybe merged into the virtual disk file. The virtualization manager 1169may perform various virtual machine-related tasks, such as cloningvirtual machines, creating new virtual machines, monitoring the state ofvirtual machines, and moving virtual machines.

The lightweight deduplication system 1171 is configured to implementefficient deduplication approaches, as discussed in further detailbelow. Although the lightweight deduplication system 1171 is illustratedas operating on the server 1160, it is appreciated that the lightweightdeduplication system 1171 may be integrated and run on other devices ofthe networked computing environment 1100, including, for example onstorage appliance 1140 or storage appliance 1170.

One embodiment of storage appliance 1170 includes a network interface1175, processor 1176, memory 1177, and disk 1178 all in communicationwith each other. Network interface 1175 allows storage appliance 1170 toconnect to one or more networks 1180. Network interface 1175 may includea wireless network interface and/or a wired network interface. Processor1176 allows storage appliance 1170 to execute instructions stored inmemory 1177 in order to perform processes described herein. Processor1176 may include one or more processing units, such as one or more CPUsand/or one or more GPUs. Memory 1177 may comprise one or more types ofmemory (e.g., RAM, SRAM, DRAM, ROM, EEPROM, NOR Flash, NAND Flash,etc.). Disk 1178 may include a hard disk drive and/or a solid-statedrive. Memory 1177 and disk 1178 may comprise hardware storage devices.

In one embodiment, the storage appliance 1170 may include four machines.Each of the four machines may include a multi-core CPU, 64 GB of RAM, a400 GB SSD, three 4 TB HDDs, and a network interface controller. In thiscase, the four machines may be in communication with the one or morenetworks 1180 via the four network interface controllers. The fourmachines may comprise four nodes of a server cluster. The server clustermay comprise a set of physical machines that are connected together viaa network. The server cluster may be used for storing data associatedwith a plurality of virtual machines, such as backup data associatedwith different point-in-time versions of a thousand virtual machines.The networked computing environment 1100 may provide a cloud computingenvironment for one or more computing devices. Cloud computing may referto Internet-based computing, wherein shared resources, software, and/orinformation may be provided to one or more computing devices on-demandvia the Internet. The networked computing environment 1100 may comprisea cloud computing environment providing Software-as-a-Service (SaaS) orInfrastructure-as-a-Service (IaaS) services. SaaS may refer to asoftware distribution model in which applications are hosted by aservice provider and made available to end users over the Internet. Inone embodiment, the networked computing environment 1100 may include avirtualized infrastructure that provides software, data processing,and/or data storage services to end users accessing the services via thenetworked computing environment 1100. In one example, networkedcomputing environment 1100 may provide cloud-based work productivity orbusiness-related applications to a computing device, such as computingdevice 1154. The storage appliance 1140 may comprise a cloud-based datamanagement system for backing up virtual machines and/or files within avirtualized infrastructure, such as virtual machines running on server1160 or files stored on server 1160 (e.g., locally stored files, filesstored in mounted directories), according to some example embodiments.

In some cases, networked computing environment 1100 may provide remoteaccess to secure applications and files stored within datacenter 1150from a remote computing device, such as computing device 1154. Thedatacenter 1150 may use an access control application to manage remoteaccess to protected resources, such as protected applications,databases, or files located within the data center. To facilitate remoteaccess to secure applications and files, a secure network connection maybe established using a virtual private network (VPN). A VPN connectionmay allow a remote computing device, such as computing device 1154, tosecurely access data from a private network (e.g., from a company fileserver or mail server) using an unsecure public network or the Internet.The VPN connection may require client-side software (e.g., running onthe remote computing device) to establish and maintain the VPNconnection. The VPN client software may provide data encryption andencapsulation prior to the transmission of secure private networktraffic through the Internet.

In some embodiments, the storage appliance 1170 may manage theextraction and storage of virtual machine snapshots associated withdifferent point-in-time versions of one or more virtual machines runningwithin the datacenter 1150. A snapshot of a virtual machine maycorrespond with a state of the virtual machine at a particular point intime. In response to a restore command from the server 1160, the storageappliance 1170 may restore a point-in-time version of a virtual machineor restore point-in-time versions of one or more files located on thevirtual machine and transmit the restored data to the server 1160. Inresponse to a mount command from the server 1160, the storage appliance1170 may allow a point-in-time version of a virtual machine to bemounted and allow the server 1160 to read and/or modify data associatedwith the point-in-time version of the virtual machine. To improvestorage density, the storage appliance 1170 may deduplicate and compressdata associated with different versions of a virtual machine and/ordeduplicate and compress data associated with different virtualmachines. To improve system performance, the storage appliance 1170 mayfirst store virtual machine snapshots received from a virtualizedenvironment in a cache, such as a flash-based cache. The cache may alsostore popular data or frequently accessed data (e.g., based on a historyof virtual machine restorations, incremental files associated withcommonly restored virtual machine versions) and current-day incrementalfiles or incremental files corresponding with snapshots captured withinthe past twenty-four hours.

An incremental file may comprise a forward incremental file or a reverseincremental file. A forward incremental file may include a set of datarepresenting changes that have occurred since an earlier point-in-timesnapshot of a virtual machine. To generate a snapshot of the virtualmachine corresponding with a forward incremental file, the forwardincremental file may be combined with an earlier point-in-time snapshotof the virtual machine (e.g., the forward incremental file may becombined with the last full image of the virtual machine that wascaptured before the forward incremental file was captured and any otherforward incremental files that were captured subsequent to the last fullimage and prior to the forward incremental file). A reverse incrementalfile may include a set of data representing changes from a laterpoint-in-time snapshot of a virtual machine. To generate a snapshot ofthe virtual machine corresponding with a reverse incremental file, thereverse incremental file may be combined with a later point-in-timesnapshot of the virtual machine (e.g., the reverse incremental file maybe combined with the most recent snapshot of the virtual machine and anyother reverse incremental files that were captured prior to the mostrecent snapshot and subsequent to the reverse incremental file).

The storage appliance 1170 may provide a user interface (e.g., aweb-based interface or a graphical user interface) that displays virtualmachine backup information such as identifications of the virtualmachines protected and the historical versions or time machine views foreach of the virtual machines protected. A time machine view of a virtualmachine may include snapshots of the virtual machine over a plurality ofpoints in time. Each snapshot may comprise the state of the virtualmachine at a particular point in time. Each snapshot may correspond witha different version of the virtual machine (e.g., Version 1 of a virtualmachine may correspond with the state of the virtual machine at a firstpoint in time and Version 2 of the virtual machine may correspond withthe state of the virtual machine at a second point in time subsequent tothe first point in time).

The user interface may enable an end user of the storage appliance 1170(e.g., a system administrator or a virtualization administrator) toselect a particular version of a virtual machine to be restored ormounted. When a particular version of a virtual machine has beenmounted, the particular version may be accessed by a client (e.g., avirtual machine, a physical machine, or a computing device) as if theparticular version was local to the client. A mounted version of avirtual machine may correspond with a mount point directory (e.g.,/snapshots/VM5Nersion23). In one example, the storage appliance 1170 mayrun a NFS server and make the particular version (or a copy of theparticular version) of the virtual machine accessible for reading and/orwriting. The end user of the storage appliance 1170 may then select theparticular version to be mounted and run an application (e.g., a dataanalytics application) using the mounted version of the virtual machine.In another example, the particular version may be mounted as an iSCSItarget.

In some example embodiments, the storage appliance 1140 is an externalnetwork-connected database appliance comprising an agent 1142, anapplication 1144, and a storage device 1146. In some exampleembodiments, the agent 1142 (e.g., backup agent 168) may be uploadedfrom the datacenter 1150 and installed on the storage appliance 1140.After installation on the storage appliance 1140, the agent 1142 may beenabled or disabled by the storage appliance 1140 over time. The agent1142 may acquire one or more electronic files or snapshot informationassociated with the one or more electronic files from the application1144. The snapshot information may include full and/or differentialsnapshot data. In one example, the one or more electronic files maycomprise a database file for a database and the snapshot information maycomprise a differential backup of the database file.

In those embodiments in which the application 1144 is a databaseapplication that manages a database, the agent 1142 is configured toacquire one or more electronic files corresponding with a firstpoint-in-time version of the database from the database application. Theagent 1142 can further acquire a database file for the database from theapplication 1144 or acquire a full or differential backup of thedatabase from the computing application 1144. The determination ofwhether the agent 1142 acquires the database file or the full ordifferential backup may depend on a file size of the database file. Thedatabase file may comprise a text file or a binary file. The agent 1142may transfer one or more changed data blocks corresponding with thefirst point-in-time version of the database to the storage appliance1140.

In some example embodiments, the agent 1142 is further configured tointerface with application 1144 or storage device 1146 to implementchanges, such as creating directories, database instances, reads/writes,and other operations to provide database management functions betweenthe storage appliance 1140 and devices within datacenter 1150. Forexample, the application 1144 can be a relational database managementapplication with plugin functionality, in which third-party developedplugins or extensions can be integrated in the application 1144 toperform actions, such as creation of a database instance.

In some example embodiments, the application 1144 is a databaseapplication for managing a database (e.g., Oracle database managementsystem) that can store database data locally on storage device 1146, oron remote storage locations, such as within datacenter 1150. The agent1142 is a remote connection system for performing snapshots of databasedata (e.g., databases managed by application 1144), and can furtherimplement bootstrapping, upgrade, and further include backup features totransfer data from the storage appliance 1140 to datacenter 1150 vianetworks 1180.

According to an embodiment, the storage appliance 1140 may be embodiedas the source host 106. In this embodiment, multiple storage appliances1140 may be clustered together to form a clustered database. Forexample, an Oracle host with a RAC cluster may be embodied in multiplestorage appliances 1140. According to an embodiment, the storageappliance 1140 may be embodied as the remote server coupled to thestorage device 117. In this embodiment, multiple storage appliances 1140may be clustered together to form a clustered database. For example, anOracle host with a RAC cluster may be embodied in multiple storageappliances 1140. According to an embodiment, the storage appliance 1170may be embodied as the backup host 104. In this embodiment, multiplestorage appliances 1170 may be clustered together to form a clustereddatabase, as previously described.

FIG. 7B depicts one embodiment of server 1160 in FIG. 7A. The server1160 may comprise one server out of a plurality of servers that arenetworked together within a data center (e.g., datacenter 1150). In oneexample, the plurality of servers may be positioned within one or moreserver racks within the data center. As depicted, the server 1160includes hardware-level components and software-level components. Thehardware-level components include one or more processors 1182, one ormore memory 1184, and one or more disks 1185. The software-levelcomponents include a hypervisor 1186, a virtualized infrastructuremanager 1199, and one or more virtual machines, such as virtual machine1198. The hypervisor 1186 may comprise a native hypervisor or a hostedhypervisor. The hypervisor 1186 may provide a virtual operating platformfor running one or more virtual machines, such as virtual machine 1198.Virtual machine 1198 includes a plurality of virtual hardware devicesincluding a virtual processor 1192, a virtual memory 1194, and a virtualdisk 1195. The virtual disk 1195 may comprise a file stored within theone or more disks 1185. In one example, a virtual machine 1198 mayinclude a plurality of virtual disks, with each virtual disk of theplurality of virtual disks associated with a different file stored onthe one or more disks 1185. Virtual machine 1198 may include a guestoperating system 1196 that runs one or more applications, such asapplication 1197.

The virtualized infrastructure manager 1199, which may correspond withthe virtualization manager 1169 in FIG. 7A, may run on a virtual machineor natively on the server 1160. The virtualized infrastructure manager1199 may provide a centralized platform for managing a virtualizedinfrastructure that includes a plurality of virtual machines. Thevirtualized infrastructure manager 1199 may manage the provisioning ofvirtual machines running within the virtualized infrastructure andprovide an interface to computing devices interacting with thevirtualized infrastructure. The virtualized infrastructure manager 1199may perform various virtualized infrastructure-related tasks, such ascloning virtual machines, creating new virtual machines, monitoring thestate of virtual machines, and facilitating backups of virtual machines.

In one embodiment, the server 1160 may use the virtualizedinfrastructure manager 1199 to facilitate backups for a plurality ofvirtual machines (e.g., eight different virtual machines) running on theserver 1160. Each virtual machine running on the server 1160 may run itsown guest operating system and its own set of applications. Each virtualmachine running on the server 1160 may store its own set of files usingone or more virtual disks associated with the virtual machine (e.g.,each virtual machine may include two virtual disks that are used forstoring data associated with the virtual machine).

In one embodiment, a data management application running on a storageappliance, such as storage appliance 1140 in FIG. 7A or storageappliance 1170 in FIG. 7A, may request a snapshot of a virtual machinerunning on the server 1160. The snapshot of the virtual machine may bestored as one or more files, with each file associated with a virtualdisk of the virtual machine. A snapshot of a virtual machine maycorrespond with a state of the virtual machine at a particular point intime. The particular point in time may be associated with a time stamp.In one example, a first snapshot of a virtual machine may correspondwith a first state of the virtual machine (including the state ofapplications and files stored on the virtual machine) at a first pointin time and a second snapshot of the virtual machine may correspond witha second state of the virtual machine at a second point in timesubsequent to the first point in time.

In response to a request for a snapshot of a virtual machine at aparticular point in time, the virtualized infrastructure manager 1199may set the virtual machine into a frozen state or store a copy of thevirtual machine at the particular point in time. The virtualizedinfrastructure manager 1199 may then transfer data associated with thevirtual machine (e.g., an image of the virtual machine or a portion ofthe image of the virtual machine) to the storage appliance. The dataassociated with the virtual machine may include a set of files includinga virtual disk file storing contents of a virtual disk of the virtualmachine at the particular point in time and a virtual machineconfiguration file storing configuration settings for the virtualmachine at the particular point in time. The contents of the virtualdisk file may include the operating system used by the virtual machine,local applications stored on the virtual disk, and user files (e.g.,images and word processing documents). In some cases, the virtualizedinfrastructure manager 1199 may transfer a full image of the virtualmachine to the storage appliance or a plurality of data blockscorresponding with the full image (e.g., to enable a full image-levelbackup of the virtual machine to be stored on the storage appliance). Inother cases, the virtualized infrastructure manager 1199 may transfer aportion of an image of the virtual machine associated with data that haschanged since an earlier point in time prior to the particular point intime or since a last snapshot of the virtual machine was taken. In oneexample, the virtualized infrastructure manager 1199 may transfer onlydata associated with virtual blocks stored on a virtual disk of thevirtual machine that have changed since the last snapshot of the virtualmachine was taken. In one embodiment, the data management applicationmay specify a first point in time and a second point in time and thevirtualized infrastructure manager 1199 may output one or more virtualdata blocks associated with the virtual machine that have been modifiedbetween the first point in time and the second point in time.

In some embodiments, the server 1160 or the hypervisor 1186 maycommunicate with a storage appliance, such as storage appliance 1140 inFIG. 7A or storage appliance 1170 in FIG. 7A, using a distributed filesystem protocol such as NFS Version 3. The distributed file systemprotocol may allow the server 1160 or the hypervisor 1186 to access,read, write, or modify files stored on the storage appliance 1140/1170as if the files were locally stored on the server 1160. The distributedfile system protocol may allow the server 1160 or the hypervisor 1186 tomount a directory or a portion of a file system located within thestorage appliance 1140 or the storage appliance 1170. For example, thestorage appliance 1140 can include a standalone host of a database,where the server 1160 mounts the database directories as if the fileswere locally stored on server 1160. Further, the server 1160 mayfunction as a backup device for storage appliance 1140 by backing updata in the mounted directories in a distributed database withindatacenter 1150, such as a cluster of nodes in storage appliance 1170.

FIG. 7C depicts one embodiment of storage appliance 1170 (e.g., serverstorage platform) in FIG. 7A. The storage appliance 1170 may include aplurality of physical machines that may be grouped together andpresented as a single computing system. Each physical machine of theplurality of physical machines may comprise a node in a cluster (e.g., afailover cluster). In one example, the storage appliance 1170 may bepositioned within a server rack within a data center. As depicted, thestorage appliance 1170 includes hardware-level components andsoftware-level components. The hardware-level components include one ormore physical machines, such as physical machine 1120 and physicalmachine 1130. The physical machine 1120 includes a network interface1121, processor 1122, memory 1123, and disk 1124 all in communicationwith each other. Processor 1122 allows physical machine 1120 to executecomputer-readable instructions stored in memory 1123 to performprocesses described herein. Disk 1124 may include a HDD and/or a SDD.The physical machine 1130 includes a network interface 1131, processor1132, memory 1133, and disk 1134 all in communication with each other.Processor 1132 allows physical machine 1130 to execute computer-readableinstructions stored in memory 1133 to perform processes describedherein. Disk 1134 may include a HDD and/or a SSD. In some cases, disk1134 may include a flash-based SSD or a hybrid HDD/SSD drive. In oneembodiment, the storage appliance 1170 may include a plurality ofphysical machines arranged in a cluster (e.g., eight machines in acluster). Each of the plurality of physical machines may include aplurality of multi-core CPUs, 128 GB of RAM, a 500 GB SSD, four 4 TBHDDs, and a network interface controller.

In some embodiments, the plurality of physical machines may be used toimplement a cluster-based network fileserver. The cluster-based networkfile server may neither require nor use a front-end load balancer. Oneissue with using a front-end load balancer to host the IP address forthe cluster-based network file server and to forward requests to thenodes of the cluster-based network file server is that the front-endload balancer comprises a single point of failure for the cluster-basednetwork file server. In some cases, the file system protocol used by aserver, such as server 1160 in FIG. 7A, or a hypervisor, such ashypervisor 1186 in FIG. 7B, to communicate with the storage appliance1170 may not provide a failover mechanism (e.g., NFS Version 3). In thecase that no failover mechanism is provided on the client side, thehypervisor may not be able to connect to a new node within a cluster inthe event that the node connected to the hypervisor fails.

In some embodiments, each node in a cluster may be connected to eachother via a network and may be associated with one or more IP addresses(e.g., two different IP addresses may be assigned to each node). In oneexample, each node in the cluster may be assigned a permanent IP addressand a floating IP address and may be accessed using either the permanentIP address or the floating IP address. In this case, a hypervisor, suchas hypervisor 1186 in FIG. 7B, may be configured with a first floatingIP address associated with a first node in the cluster. The hypervisormay connect to the cluster using the first floating IP address. In oneexample, the hypervisor may communicate with the cluster using the NFSVersion 3 protocol.

Each node in the cluster may run a Virtual Router Redundancy Protocol(VRRP) daemon. A daemon may comprise a background process. Each VRRPdaemon may include a list of all floating IP addresses available withinthe cluster. In the event that the first node associated with the firstfloating IP address fails, one of the VRRP daemons may automaticallyassume or pick up the first floating IP address if no other VRRP daemonhas already assumed the first floating IP address. Therefore, if thefirst node in the cluster fails or otherwise goes down, then one of theremaining VRRP daemons running on the other nodes in the cluster mayassume the first floating IP address that is used by the hypervisor forcommunicating with the cluster.

In order to determine which of the other nodes in the cluster willassume the first floating IP address, a VRRP priority may beestablished. In one example, given a number (N) of nodes in a clusterfrom node(0) to node(N−1), for a floating IP address (i), the VRRPpriority of nodeG) may be (G-i) modulo N. In another example, given anumber (N) of nodes in a cluster from node(0) to node(N−1), for afloating IP address (i), the VRRP priority of nodeG) may be (i j) moduloN. In these cases, nodeG) will assume floating IP address (i) only ifits VRRP priority is higher than that of any other node in the clusterthat is alive and announcing itself on the network. Thus, if a nodefails, there may be a clear priority ordering for determining whichother node in the cluster will take over the failed node's floating IPaddress.

In some cases, a cluster may include a plurality of nodes and each nodeof the plurality of nodes may be assigned a different floating IPaddress. In this case, a first hypervisor may be configured with a firstfloating IP address associated with a first node in the cluster, asecond hypervisor may be configured with a second floating IP addressassociated with a second node in the cluster, and a third hypervisor maybe configured with a third floating IP address associated with a thirdnode in the cluster.

As depicted in FIG. 7C, the software-level components of the storageappliance 1170 may include data management system 1102, a virtualizationinterface 1104, a distributed job scheduler 1108, a distributed metadatastore 1110, a distributed file system 1112, and one or more virtualmachine search indexes, such as virtual machine search index 1106. Inone embodiment, the software-level components of the storage appliance1170 may be run using a dedicated hardware-based appliance. In anotherembodiment, the software-level components of the storage appliance 1170may be run from the cloud (e.g., the software-level components may beinstalled on a cloud service provider).

In some cases, the data storage across a plurality of nodes in a cluster(e.g., the data storage available from the one or more physicalmachines) may be aggregated and made available over a single file systemnamespace (e.g., /snap-50 shots/). A directory for each virtual machineprotected using the storage appliance 1170 may be created (e.g., thedirectory for Virtual Machine A may be /snapshots/VM_A). Snapshots andother data associated with a virtual machine may reside within thedirectory for the virtual machine. In one example, snapshots of avirtual machine may be stored in subdirectories of the directory (e.g.,a first snapshot of Virtual Machine A may reside in /snapshots/VM_A/s1/and a second snapshot of Virtual Machine A may reside in/snapshots/VM_A/s2/).

The distributed file system 1112 may present itself as a single filesystem, in which, as new physical machines or nodes are added to thestorage appliance 1170, the cluster may automatically discover theadditional nodes and automatically increase the available capacity ofthe file system for storing files and other data. Each file stored inthe distributed file system 1112 may be partitioned into one or morechunks or shards. Each of the one or more chunks may be stored withinthe distributed file system 1112 as a separate file. The files storedwithin the distributed file system 1112 may be replicated or mirroredover a plurality of physical machines, thereby creating a load-balancedand fault-tolerant distributed file system. In one example, storageappliance 1170 may include ten physical machines arranged as a failovercluster and a first file corresponding with a snapshot of a virtualmachine (e.g., /snapshots/VM_A/s1/s1.full) may be replicated and storedon three of the ten machines.

The distributed metadata store 1110 may include a distributed databasemanagement system that provides high availability without a single pointof failure. In one embodiment, the distributed metadata store 1110 maycomprise a database, such as a distributed document-oriented database.The distributed metadata store 1110 may be used as a distributed keyvalue storage system. In one example, the distributed metadata store1110 may comprise a distributed NoSQL key value store database. In somecases, the distributed metadata store 1110 may include a partitioned rowstore, in which rows are organized into tables or other collections ofrelated data held within a structured format within the key value storedatabase. A table (or a set of tables) may be used to store metadatainformation associated with one or more files stored within thedistributed file system 1112. The metadata information may include ametadata item 166, the name of a file, a size of the file, filepermissions associated with the file, when the file was last modified,and file mapping information associated with an identification of thelocation of the file stored within a cluster of physical machines. Inone embodiment, a new file corresponding with a snapshot of a virtualmachine may be stored within the distributed file system 1112 andmetadata associated with the new file may be stored within thedistributed metadata store 1110. The distributed metadata store 1110 mayalso be used to store a backup schedule for the virtual machine and alist of snapshots for the virtual machine that are stored using thestorage appliance 1170.

In some cases, the distributed metadata store 1110 may be used to manageone or more versions of a virtual machine. Each version of the virtualmachine may correspond with a full image snapshot of the virtual machinestored within the distributed file system 1112 or an incrementalsnapshot of the virtual machine (e.g., a forward incremental or reverseincremental) stored within the distributed file system 1112. In oneembodiment, the one or more versions of the virtual machine maycorrespond with a plurality of files. The plurality of files may includea single full image snapshot of the virtual machine and one or moreincrementals derived from the single full image snapshot. The singlefull image snapshot of the virtual machine may be stored using a firststorage device of a first type (e.g., a HDD) and the one or moreincrementals derived from the single full image snapshot may be storedusing a second storage device of a second type (e.g., an SSD). In thiscase, only a single full image needs to be stored and each version ofthe virtual machine may be generated from the single full image or thesingle full image combined with a subset of the one or moreincrementals. Furthermore, each version of the virtual machine may begenerated by performing a sequential read from the first storage device(e.g., reading a single file from a HDD) to acquire the full image and,in parallel, performing one or more reads from the second storage device(e.g., performing fast random reads from an SSD) to acquire the one ormore incrementals.

The distributed job scheduler 1108 (e.g., job module 152) may be usedfor scheduling backup jobs that acquire and store virtual machinesnapshots for one or more virtual machines over time. The distributedjob scheduler 1108 may follow a backup schedule to back up an entireimage of a virtual machine at a particular point in time or one or morevirtual disks associated with the virtual machine at the particularpoint in time. In one example, the backup schedule may specify that thevirtual machine be backed up at a snapshot capture frequency, such asevery two hours or every 24 hours. Each backup job may be associatedwith one or more tasks to be performed in a sequence. Each of the one ormore tasks associated with a job may be run on a particular node withina cluster. In some cases, the distributed job scheduler 1108 mayschedule a specific job to be run on a particular node based on datastored on the particular node. For example, the distributed jobscheduler 1108 may schedule a virtual machine snapshot job to be run ona node in a cluster that is used to store snapshots of the virtualmachine in order to reduce network congestion.

The distributed job scheduler 1108 may comprise a distributedfault-tolerant job scheduler, in which jobs affected by node failuresare recovered and rescheduled to be run on available nodes. In oneembodiment, the distributed job scheduler 1108 may be fullydecentralized and implemented without the existence of a master node.The distributed job scheduler 1108 may run job scheduling processes oneach node in a cluster or on a plurality of nodes in the cluster. In oneexample, the distributed job scheduler 1108 may run a first set of jobscheduling processes on a first node in the cluster, a second set of jobscheduling processes on a second node in the cluster, and a third set ofjob scheduling processes on a third node in the cluster. The first setof job scheduling processes, the second set of job scheduling processes,and the third set of job scheduling processes may store informationregarding jobs, schedules, and the states of jobs using a metadatastore, such as distributed metadata store 1110. In the event that thefirst node running the first set of job scheduling processes fails(e.g., due to a network failure or a physical machine failure), thestates of the jobs managed by the first set of job scheduling processesmay fail to be updated within a threshold period of time (e.g., a jobmay fail to be completed within 30 seconds or within minutes from beingstarted). In response to detecting jobs that have failed to be updatedwithin the threshold period of time, the distributed job scheduler 1108may undo and restart the failed jobs on available nodes within thecluster.

The job scheduling processes running on at least a plurality of nodes ina cluster (e.g., on each available node in the cluster) may manage thescheduling and execution of a plurality of jobs. The job schedulingprocesses may include run processes for running jobs, cleanup processesfor cleaning up failed tasks, and rollback processes for rolling back orundoing any actions or tasks performed by failed jobs. In oneembodiment, the job scheduling processes may detect that a particulartask for a particular job has failed and, in response, may perform acleanup process to clean up or remove the effects of the particular taskand then perform a rollback process that processes one or more completedtasks for the particular job in reverse order to undo the effects of theone or more completed tasks. Once the particular job with the failedtask has been undone, the job scheduling processes may restart theparticular job on an available node in the cluster.

The distributed job scheduler 1108 may manage a job in which a series oftasks associated with the job are to be performed atomically (i.e.,partial execution of the series of tasks is not permitted). If theseries of tasks cannot be completely executed or there is any failurethat occurs to one of the series of tasks during execution (e.g., a harddisk associated with a physical machine fails or a network connection tothe physical machine fails), then the state of a data management systemmay be returned to a state as if none of the series of tasks were everperformed. The series of tasks may correspond with an ordering of tasksfor the series of tasks and the distributed job scheduler 1108 mayensure that each task of the series of tasks is executed based on theordering of tasks. Tasks that do not have dependencies with each othermay be executed in parallel.

In some cases, the distributed job scheduler 1108 may schedule each taskof a series of tasks to be performed on a specific node in a cluster. Inother cases, the distributed job scheduler 1108 may schedule a firsttask of the series of tasks to be performed on a first node in a clusterand a second task of the series of tasks to be performed on a secondnode in the cluster. In these cases, the first task may have to operateon a first set of data (e.g., a first file stored in a file system)stored on the first node and the second task may have to operate on asecond set of data (e.g., metadata related to the first file that isstored in a database) stored on the second node. In some embodiments,one or more tasks associated with a job may have an affinity to aspecific node in a cluster.

In one example, if the one or more tasks require access to a databasethat has been replicated on three nodes in a cluster, then the one ormore tasks may be executed on one of the three nodes. In anotherexample, if the one or more tasks require access to multiple chunks ofdata associated with a virtual disk that has been replicated over fournodes in a cluster, then the one or more tasks may be executed on one ofthe four nodes. Thus, the distributed job scheduler 1108 may assign oneor more tasks associated with a job to be executed on a particular nodein a cluster based on the location of data required to be accessed bythe one or more tasks.

In one embodiment, the distributed job scheduler 1108 may manage a firstjob associated with capturing and storing a snapshot of a virtualmachine periodically (e.g., every 30 minutes). The first job may includeone or more tasks, such as communicating with a virtualizedinfrastructure manager, such as the virtualized infrastructure manager1199 in FIG. 7B, to create a frozen copy of the virtual machine and totransfer one or more chunks (or one or more files) associated with thefrozen copy to a storage appliance, such as storage appliance 1170 inFIG. 7A. The one or more tasks may also include generating metadata forthe one or more chunks, storing the metadata using the distributedmetadata store 1110, storing the one or more chunks within thedistributed file system 1112, and communicating with the virtualizedinfrastructure manager that the frozen copy of the virtual machine maybe unfrozen or released from a frozen state. The metadata for a firstchunk of the one or more chunks may include information specifying aversion of the virtual machine associated with the frozen copy, a timeassociated with the version (e.g., the snapshot of the virtual machinewas taken at 5:30 p.m. on Jun. 29, 2018), and a file path to where thefirst chunk is stored within the distributed file system 1112 (e.g., thefirst chunk is located at /snapshotsNM_B/s1/s1.chunk1). The one or moretasks may also include deduplication, compression (e.g., using alossless data compression algorithm such as LZ4 or LZ77), decompression,encryption (e.g., using a symmetric key algorithm such as Triple DES orAES-256), and decryption-related tasks.

The virtualization interface 1104 may provide an interface forcommunicating with a virtualized infrastructure manager managing avirtualization infrastructure, such as virtualized infrastructuremanager 1199 in FIG. 7B, and requesting data associated with virtualmachine snapshots from the virtualization infrastructure. Thevirtualization interface 1104 may communicate with the virtualizedinfrastructure manager using an API for accessing the virtualizedinfrastructure manager (e.g., to communicate a request for a snapshot ofa virtual machine). In this case, storage appliance 1170 may request andreceive data from a virtualized infrastructure without requiring agentsoftware to be installed or running on virtual machines within thevirtualized infrastructure. The virtualization interface 1104 mayrequest data associated with virtual blocks stored on a virtual disk ofthe virtual machine that have changed since a last snapshot of thevirtual machine was taken or since a specified prior point in time.Therefore, in some cases, if a snapshot of a virtual machine is thefirst snapshot taken of the virtual machine, then a full image of thevirtual machine may be transferred to the storage appliance. However, ifthe snapshot of the virtual machine is not the first snapshot taken ofthe virtual machine, then only the data blocks of the virtual machinethat have changed since a prior snapshot was taken may be transferred tothe storage appliance.

The virtual machine search index 1106 may include a list of files thathave been stored using a virtual machine and a version history for eachof the files in the list. Each version of a file may be mapped to theearliest point-in-time snapshot of the virtual machine that includes theversion of the file or to a snapshot of the virtual machine thatincludes the version of the file (e.g., the latest point-in-timesnapshot of the virtual machine that includes the version of the file).In one example, the virtual machine search index 1106 may be used toidentify a version of the virtual machine that includes a particularversion of a file (e.g., a particular version of a database, aspreadsheet, or a word processing document). In some cases, each of thevirtual machines that are backed up or protected using storage appliance1170 may have a corresponding virtual machine search index.

In one embodiment, as each snapshot of a virtual machine is ingested,each virtual disk associated with the virtual machine is parsed in orderto identify a file system type associated with the virtual disk and toextract metadata (e.g., file system metadata) for each file stored onthe virtual disk. The metadata may include information for locating andretrieving each file from the virtual disk. The metadata may alsoinclude a name of a file, the size of the file, the last time at whichthe file was modified, and a content checksum for the file. Each filethat has been added, deleted, or modified since a previous snapshot wascaptured may be determined using the metadata (e.g., by comparing thetime at which a file was last modified with a time associated with theprevious snapshot). Thus, for every file that has existed within any ofthe snapshots of the virtual machine, a virtual machine search index maybe used to identify when the file was first created (e.g., correspondingwith a first version of the file) and at what times the file wasmodified (e.g., corresponding with subsequent versions of the file).Each version of the file may be mapped to a particular version of thevirtual machine that stores that version of the file.

In some cases, if a virtual machine includes a plurality of virtualdisks, then a virtual machine search index may be generated for eachvirtual disk of the plurality of virtual disks. For example, a firstvirtual machine search index may catalog and map files located on afirst virtual disk of the plurality of virtual disks and a secondvirtual machine search index may catalog and map files located on asecond virtual disk of the plurality of virtual disks. In this case, aglobal file catalog or a global virtual machine search index for thevirtual machine may include the first virtual machine search index andthe second virtual machine search index. A global file catalog may bestored for each virtual machine backed up by a storage appliance withina file system, such as distributed file system 1112 in FIG. 7C. The datamanagement system 1102 (e.g., processing module 154) may comprise anapplication running on the storage appliance that manages and stores oneor more snapshots of a virtual machine. In one example, the datamanagement system 1102 may comprise a highest-level layer in anintegrated software stack running on the storage appliance. Theintegrated software stack may include the data management system 1102,the virtualization interface 1104, the distributed job scheduler 1108,the distributed metadata store 1110, and the distributed file system1112.

In some cases, the integrated software stack may run on other computingdevices, such as a server or computing device 1154 in FIG. 7A. The datamanagement system 1102 may use the virtualization interface 1104, thedistributed job scheduler 1108, the distributed metadata store 1110, andthe distributed file system 1112 to manage and store one or moresnapshots of a virtual machine. Each snapshot of the virtual machine maycorrespond with a point-in-time version of the virtual machine. The datamanagement system 1102 may generate and manage a list of versions forthe virtual machine. Each version of the virtual machine may map to orreference one or more chunks and/or one or more files stored within thedistributed file system 1112. Combined together, the one or more chunksand/or the one or more files stored within the distributed file system1112 may comprise a full image of the version of the virtual machine.

The modules, methods, engines, applications, and so forth described inconjunction with FIG. 1A-FIG. 6 are implemented in some embodiments inthe context of multiple machines and associated software architectures.The sections below describe representative software architecture(s) andmachine (e.g., hardware) architecture(s) that are suitable for use withthe disclosed embodiment

Software architectures are used in conjunction with hardwarearchitectures to create devices and machines tailored to particularpurposes. For example, a particular hardware architecture coupled with aparticular software architecture will create a mobile device, such as amobile phone, tablet device, or so forth. A slightly different hardwareand software architecture may yield a smart device for use in the“Internet of Things,” while yet another combination produces a servercomputer for use within a cloud computing architecture. Not allcombinations of such software and hardware architectures are presentedhere, as those of skill in the art can readily understand how toimplement the disclosure in different contexts from the disclosurecontained herein.

FIG. 7D shows an example cluster 1200 of a distributed decentralizeddatabase, according to some example embodiments. As illustrated, theexample cluster 1200 includes five nodes, nodes 1-5. In some exampleembodiments, each of the five nodes runs from different machines, suchas physical machine 1120 in FIG. 7C or virtual machine 1198 in FIG. 7B.The nodes in the cluster 1200 can include instances of peer nodes of adistributed database (e.g., cluster-based database, distributeddecentralized database management system, a NoSQL database, ApacheCassandra, DataStax, MongoDB, CouchDB), according to some exampleembodiments. The distributed database system is distributed in that datais sharded or distributed across the cluster 1200 in shards or chunksand decentralized in that there is no central storage device andtherefore no single point of failure. The system operates under anassumption that multiple nodes may go down, up, or become nonresponsive,and so on. Sharding is splitting up of the data horizontally andmanaging each separately on different nodes. For example, if the datamanaged by the cluster 1200 can be indexed using the 26 letters of thealphabet, node 1 can manage a first shard that handles records thatstart with A through E, node 2 can manage a second shard that handlesrecords that start with F through J, and so on.

In some example embodiments, data written to one of the nodes isreplicated to one or more other nodes per a replication protocol of thecluster 1200. For example, data written to node 1 can be replicated tonodes 2 and 3. If node 1 prematurely terminates, node 2 and/or 3 can beused to provide the replicated data. In some example embodiments, eachnode of cluster 1200 frequently exchanges state information about itselfand other nodes across the cluster 1200 using gossip protocol. Gossipprotocol is a peer-to-peer communication protocol in which each noderandomly shares (e.g., communicates, requests, transmits) location andstate information about the other nodes in a given cluster.

Writing: For a given node, a sequentially written commit log capturesthe write activity to ensure data durability. The data is then writtento an in-memory structure (e.g., a memtable, write-back cache). Eachtime the in-memory structure is full, the data is written to disk in aSorted String Table data file. In some example embodiments, writes areautomatically partitioned and replicated throughout the cluster 1200.

Reading: Any node of cluster 1200 can receive a read request (e.g.,query) from an external client. If the node that receives the readrequest manages the data requested, the node provides the requesteddata. If the node does not manage the data, the node determines whichnode manages the requested data. The node that received the read requestthen acts as a proxy between the requesting entity and the node thatmanages the data (e.g., the node that manages the data sends the data tothe proxy node, which then provides the data to an external entity thatgenerated the request).

The distributed decentralized database system is decentralized in thatthere is no single point of failure due to the nodes being symmetricaland seamlessly replaceable. For example, whereas conventionaldistributed data implementations have nodes with different functions(e.g., master/slave nodes, asymmetrical database nodes, federateddatabases), the nodes of cluster 1200 are configured to function thesame way (e.g., as symmetrical peer database nodes that communicate viagossip protocol, such as Cassandra nodes) with no single point offailure. If one of the nodes in cluster 1200 terminates prematurely(“goes down”), another node can rapidly take the place of the terminatednode without disrupting service. The cluster 1200 can be a container fora keyspace, which is a container for data in the distributeddecentralized database system (e.g., whereas a database is a containerfor containers in conventional relational databases, the Cassandrakeyspace is a container for a Cassandra database system).

FIG. 8 is a block diagram 2000 illustrating a representative softwarearchitecture 2002, which may be used in conjunction with varioushardware architectures herein described. FIG. 8 is merely a non-limitingexample of a software architecture 2002, and it will be appreciated thatmany other architectures may be implemented to facilitate thefunctionality described herein. The software architecture 2002 may beexecuting on hardware such as a machine 2100 of FIG. 9 that includes,among other things, processors 2110, memory/storage 2130, and I/Ocomponents 2150. Returning to FIG. 8, a representative hardware layer2004 is illustrated and can represent, for example, the machine 2100 ofFIG. 9. The representative hardware layer 2004 comprises one or moreprocessing units 2006 having associated executable instructions 2008.The executable instructions 2008 represent the executable instructionsof the software architecture 2002, including implementation of themethods, engines, modules, and so forth of FIG. 1A-FIG. 6. The hardwarelayer 2004 also includes memory and/or storage modules 2010, which alsohave the executable instructions 2008. The hardware layer 2004 may alsocomprise other hardware 2012, which represents any other hardware of thehardware layer 2004, such as the other hardware 2012 illustrated as partof the machine 2100.

In the example architecture of FIG. 8, the software architecture 2002may be conceptualized as a stack of layers where each layer providesparticular functionality. For example, the software architecture 2002may include layers such as an operating system 2014, libraries 2016,frameworks 2018 (e.g., middleware), applications 2020, and apresentation layer 2044. Operationally, the applications 2020 and/orother components within the layers may invoke API calls 2024 through thesoftware stack and receive a response, returned values, and so forth,illustrated as messages 2026, in response to the API calls 2024. Thelayers illustrated are representative in nature, and not all softwarearchitectures have all layers. For example, some mobile orspecial-purpose operating systems 2014 may not provide a framework 2018layer, while others may provide such a layer. Other softwarearchitectures may include additional or different layers.

The operating system 2014 may manage hardware resources and providecommon services. The operating system 2014 may include, for example, akernel 2028, services 2030, and drivers 2032. The kernel 2028 may act asan abstraction layer between the hardware and the other software layers.For example, the kernel 2028 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 2030 may provideother common services for the other software layers. The drivers 2032may be responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 2032 may include display drivers,camera drivers, Bluetooth® drivers, flash memory drivers, serialcommunication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi®drivers, audio drivers, power management drivers, and so forth dependingon the hardware configuration.

The libraries 2016 may provide a common infrastructure that may beutilized by the applications 2020 and/or other components and/or layers.The libraries 2016 typically provide functionality that allows othersoftware modules to perform tasks in an easier fashion than to interfacedirectly with the underlying operating system 2014 functionality (e.g.,kernel 2028, services 2030, and/or drivers 2032). The libraries 2016 mayinclude system libraries 2034 (e.g., C standard library) that mayprovide functions such as memory allocation functions, stringmanipulation functions, mathematic functions, and the like. In addition,the libraries 2016 may include API libraries 2036 such as medialibraries (e.g., libraries to support presentation and manipulation ofvarious media formats such as moving picture experts group (MPEG) 4,H.264, MPEG-1 or MPEG-2 Audio Layer (MP3), AAC, AMR, joint photographyexperts group (JPG), or portable network graphics (PNG)), graphicslibraries (e.g., an Open Graphics Library (OpenGL) framework that may beused to render 2D and 3D graphic content on a display), databaselibraries (e.g., Structured Query Language (SQL), SQLite that mayprovide various relational database functions), web libraries (e.g.,WebKit that may provide web browsing functionality), and the like. Thelibraries 2016 may also include a wide variety of other libraries 2038to provide many other APIs to the applications 2020 and other softwarecomponents/modules.

The frameworks 2018 (also sometimes referred to as middleware) mayprovide a higher-level common infrastructure that may be utilized by theapplications 2020 and/or other software components/modules. For example,the frameworks 2018 (e.g., middleware) may provide various graphicaluser interface functions, high-level resource management, high-levellocation services, and so forth. The frameworks 2018 may provide a broadspectrum of other APIs that may be utilized by the applications 2020and/or other software components/modules, some of which may be specificto a particular operating system 2014 or platform.

The applications 2020 include built-in applications 2040 and/orthird-party applications 2042. Examples of representative built-inapplications 2040 may include, but are not limited to, a contactsapplication, a browser application, a book reader application, alocation application, a media application, a messaging application,and/or a game application. Third-party applications 2042 may include anyof the built-in applications as well as a broad assortment of otherapplications 2020. In a specific example, the third-party application2042 (e.g., an application developed using the Android™ or iOS™ softwaredevelopment kit (SDK) by an entity other than the vendor of theparticular platform) may be mobile software running on a mobileoperating system 2014 such as iOS™, Android™, Windows® Phone, or othermobile operating systems 2014. In this example, the third-partyapplication 2042 may invoke the API calls 2024 provided by the mobileoperating system such as the operating system 2014 to facilitatefunctionality described herein.

The applications 2020 may utilize built-in operating system functions(e.g., kernel 2028, services 2030, and/or drivers 2032), libraries(e.g., system libraries 2034, API libraries 2036, and other libraries2038), and frameworks 2018 to create user interfaces to interact withusers of the system. Alternatively, or additionally, in some systems,interactions with a user may occur through a presentation layer, such asthe presentation layer 2044. In these systems, the application/module“logic” can be separated from the aspects of the application/module thatinteract with a user.

Some software architectures 2002 utilize virtual machines. In theexample of FIG. 8, this is illustrated by a virtual machine 2048. Thevirtual machine 2048 creates a software environment whereapplications/modules can execute as if they were executing on a hardwaremachine (such as the machine 2100 of FIG. 9, for example). The virtualmachine 2048 is hosted by a host operating system (e.g., operatingsystem 2014 in FIG. 8) and typically, although not always, has a virtualmachine monitor 2046, which manages the operation of the virtual machine2048 as well as the interface with the host operating system (e.g.,operating system 2014). A software architecture executes within thevirtual machine 2048, such as an operating system 2050, libraries 2052,frameworks/middleware 2054, applications 2056, and/or a presentationlayer 2058. These layers of software architecture executing within thevirtual machine 2048 can be the same as corresponding layers previouslydescribed or may be different.

FIG. 9 is a block diagram illustrating components of a machine 2100,according to some example embodiments, able to read instructions from amachine-storage medium and perform any one or more of the methodologiesdiscussed herein. Specifically, FIG. 9 shows a diagrammaticrepresentation of the machine 2100 in the example form of a computersystem, within which instructions 2116 (e.g., software, a program, anapplication, an applet, an app, or other executable code) for causingthe machine 2100 to perform any one or more of the methodologiesdiscussed herein may be executed. For example, the instructions 2116 maycause the machine 2100 to execute the flow diagrams of FIG. 3A-FIG. 6.Additionally, or alternatively, the instructions 2116 may implement themodules, engines, applications, and so forth, as described in thisdocument. The instructions 2116 transform the general, non-programmedmachine 2100 into a particular machine 2100 programmed to carry out thedescribed and illustrated functions in the manner described. Inalternative embodiments, the machine 2100 operates as a standalonedevice or may be coupled (e.g., networked) to other machines 2100. In anetworked deployment, the machine 2100 may operate in the capacity of aserver machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 2100 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a set-top box (STB), apersonal digital assistant (PDA), an entertainment media system, acellular telephone, a smart phone, a mobile device, a wearable device(e.g., a smart watch), a smart home device (e.g., a smart appliance),other smart devices, a web appliance, a network router, a networkswitch, a network bridge, or any machine 2100 capable of executing theinstructions 2116, sequentially or otherwise, that specify actions to betaken by the machine 2100. Further, while only a single machine 2100 isillustrated, the term “machine” shall also be taken to include acollection of machines 2100 that individually or jointly execute theinstructions 2116 to perform any one or more of the methodologiesdiscussed herein.

The machine 2100 may include processors 2110, memory/storage 2130, andI/O components 2150, which may be configured to communicate with eachother such as via a bus 2102. In an example embodiment, the processors2110 (e.g., a CPU, a reduced instruction set computing (RISC) processor,a complex instruction set computing (CISC) processor, a GPU, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a radio-frequency integrated circuit (RFIC), another processor,or any suitable combination thereof) may include, for example, aprocessor 2112 and a processor 2114 that may execute the instructions2116. The term “processor” is intended to include multi-core processors2110 that may comprise two or more independent processors 2110(sometimes referred to as “cores”) that may execute the instructions2116 contemporaneously. Although FIG. 9 shows multiple processors 2110,the machine 2100 may include a single processor 2110 with a single core,a single processor 2110 with multiple cores (e.g., a multi-coreprocessor), multiple processors 2110 with a single core, multipleprocessors 2110 with multiples cores, or any combination thereof.

The memory/storage 2130 may include a memory 2132, such as a mainmemory, or other memory storage, and a storage unit 2136, bothaccessible to the processors 2110 such as via the bus 2102. The storageunit 2136 and memory 2132 store the instructions 2116, embodying any oneor more of the methodologies or functions described herein. Theinstructions 2116 may also reside, completely or partially, within thememory 2132, within the storage unit 2136, within at least one of theprocessors 2110 (e.g., within the processor's cache memory), or anysuitable combination thereof, during execution thereof by the machine2100. Accordingly, the memory 2132, the storage unit 2136, and thememory of the processors 2110 are examples of machine-storage media.

As used herein, “machine-storage medium” means a device able to storethe instructions 2116 and data temporarily or permanently and mayinclude, but not be limited to, RAM, ROM, buffer memory, flash memory,optical media, magnetic media, cache memory, other types of storage(e.g., EEPROM), and/or any suitable combination thereof. The term“machine-storage medium” should be taken to include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) able to store the instructions 2116. Theterm “machine-storage medium” shall also be taken to include any medium,or combination of multiple media, that is capable of storinginstructions (e.g., instructions 2116) for execution by a machine (e.g.,machine 2100), such that the instructions 2116, when executed by one ormore processors of the machine (e.g., processors 2110), cause themachine to perform any one or more of the methodologies describedherein. Accordingly, a “machine-storage medium” refers to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-storage medium” excludes signals per se.

The I/O components 2150 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 2150 that are included in a particular machine 2100 willdepend on the type of machine. For example, portable machines 2100 suchas mobile phones will likely include a touch input device or other suchinput mechanisms, while a headless server machine will likely notinclude such a touch input device. It will be appreciated that the I/Ocomponents 2150 may include many other components that are not shown inFIG. 9. The I/O components 2150 are grouped according to functionalitymerely for simplifying the following discussion and the grouping is inno way limiting. In various example embodiments, the I/O components 2150may include output components 2152 and input components 2154. The outputcomponents 2152 may include visual components (e.g., a display such as aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), haptic components (e.g., avibratory motor, resistance mechanisms), other signal generators, and soforth. The input components 2154 may include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or another pointinginstrument), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

In further example embodiments, the I/O components 2150 may includebiometric components 2156, motion components 2158, environmentalcomponents 2160, or position components 2162 among a wide array of othercomponents. For example, the biometric components 2156 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram basedidentification), and the like. The motion components 2158 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environmental components 2160 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gassensors to detect concentrations of hazardous gases for safety or tomeasure pollutants in the atmosphere), or other components that mayprovide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 2162 mayinclude location sensor components (e.g., a Global Position System (GPS)receiver component), altitude sensor components (e.g., altimeters orbarometers that detect air pressure from which altitude may be derived),orientation sensor components (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 2150 may include communication components 2164operable to couple the machine 2100 to a network 2180 or devices 2170via a coupling 2182 and a coupling 2172 respectively. For example, thecommunication components 2164 may include a network interface componentor other suitable device to interface with the network 2180. In furtherexamples, the communication components 2164 may include wiredcommunication components, wireless communication components, cellularcommunication components, near field communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 2170 may be another machine 2100 or any of awide variety of peripheral devices (e.g., a peripheral device coupledvia a USB).

Moreover, the communication components 2164 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 2164 may include radio frequency identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components2164, such as location via IP geolocation, location via Wi-Fi® signaltriangulation, location via detecting an NFC beacon signal that mayindicate a particular location, and so forth.

In various example embodiments, one or more portions of the network 2180may be an ad hoc network, an intranet, an extranet, a VPN, a LAN, awireless LAN (WLAN), a WAN, a wireless WAN (WWAN), a metropolitan areanetwork (MAN), the Internet, a portion of the Internet, a portion of thepublic switched telephone network (PSTN), a plain old telephone service(POTS) network, a cellular telephone network, a wireless network, aWi-Fi® network, another type of network, or a combination of two or moresuch networks. For example, the network 2180 or a portion of the network2180 may include a wireless or cellular network and the coupling 2182may be a Code Division Multiple Access (CDMA) connection, a GlobalSystem for Mobile communications (GSM) connection, or another type ofcellular or wireless coupling. In this example, the coupling 2182 mayimplement any of a variety of types of data transfer technology, such asSingle Carrier Radio Transmission Technology (1×RTT), Evolution-DataOptimized (EVDO) technology, General Packet Radio Service (GPRS)technology, Enhanced Data rates for GSM Evolution (EDGE) technology,third Generation Partnership Project (3GPP) including 3G, fourthgeneration wireless (4G) networks, Universal Mobile TelecommunicationsSystem (UMTS), High Speed Packet Access (HSPA), WorldwideInteroperability for Microwave Access (WiMAX), Long Term Evolution (LTE)standard, others defined by various standard-setting organizations,other long range protocols, or other data transfer technology.

The instructions 2116 may be transmitted or received over the network2180 using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components2164) and utilizing any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions2116 may be transmitted or received using a transmission medium via thecoupling 2172 (e.g., a peer-to-peer coupling) to the devices 2170.

The term “signal medium” or “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding, orcarrying the instructions 2116 for execution by the machine 2100, andincludes digital or analog communications signals or other intangiblemedia to facilitate communication of such software.

The terms “machine-readable medium,” “computer-readable medium,” and“device-readable medium” mean the same thing and may be usedinterchangeably in this disclosure. The terms are defined to includeboth machine-storage media and transmission medium. Thus, the termsinclude both storage devices/media and carrier waves/modulated datasignals.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Although an overview of the inventive subject matter has been describedwith reference to specific example embodiments, various modificationsand changes may be made to these embodiments without departing from thebroader scope of embodiments of the present disclosure. Such embodimentsof the inventive subject matter may be referred to herein, individuallyor collectively, by the term “invention” merely for convenience andwithout intending to voluntarily limit the scope of this application toany single invention or inventive concept if more than one is, in fact,disclosed.

The embodiments illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other embodiments may be used and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. The Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Moreover, plural instances may be provided forresources, operations, or structures described herein as a singleinstance. Additionally, boundaries between various resources,operations, modules, engines, and data stores are somewhat arbitrary,and particular operations are illustrated in a context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within a scope of various embodiments of thepresent disclosure. In general, structures and functionality presentedas separate resources in the example configurations may be implementedas a combined structure or resource. Similarly, structures andfunctionality presented as a single resource may be implemented asseparate resources. These and other variations, modifications,additions, and improvements fall within a scope of embodiments of thepresent disclosure as represented by the appended claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A system comprising: at least one processor andmemory having instructions that, when executed, cause the at least oneprocessor to perform operations comprising: executing a job, at a backuphost, to back up a file set from a source host responsive to atriggering event, the backing up of the file set including fetchingmetadata from the source host; identifying a first operation set from aplurality of operation sets responsive to identifying a file sharingprotocol being utilized by the backup and the source host, wherein theoperation set includes a first operation; communicating, in parallel,one or more requests, over a network, to the source host by utilizingone or more threads from a thread pool, the communicating the one ormore requests including communicating a first request, over the network,to the source host by utilizing a first thread to fetch a first metadataitem; receiving responses, over the network, corresponding to the one ormore requests by utilizing the one or more threads, the receiving theresponses including receiving a first response corresponding to thefirst request by utilizing the first thread, the first responseincluding the first metadata item; processing the responses includingprocessing the first response, comprising: generating a first latencybased on the first response, and incrementing a number of requests basedon the first response; aggregating samples responsive to a timeout, thesamples including the first latency and the number of requests; resizingthe thread pool based on the aggregating; and backing up the file setfrom the source host based on the metadata.
 2. The system of claim 1,wherein the file sharing protocol is a Network File System protocol andthe first operation set is a Unix operation set.
 3. The system of claim1, wherein the file sharing protocol is Server Message Block protocoland the first operation set is a Windows operation set.
 4. The system ofclaim 3, wherein the first request includes the first operation, forexecution on the source host, to generate the first metadata item. 5.The system of claim 1, wherein the generating the first latency furthercomprises: identifying a receive time responsive to the first threadreceiving the first response; subtracting a transmit time from thereceive time to generate the first latency; and storing the firstlatency in a first sample in a first moving average including a firstplurality of samples.
 6. The system of claim 5, wherein the aggregatingthe samples further comprises: aggregating the first latency over thefirst plurality of samples to generate a first average latency ofrequests; storing the first average latency of requests in a firstsample in a second moving average including a second plurality ofsamples; aggregating the number of requests over the first plurality ofsamples to generate a first average number of requests; and storing thefirst average number of requests in the first sample in the secondmoving average including the second plurality of samples.
 7. The systemof claim 6, wherein the resizing further comprises: computing an averagelatency ratio based on the first average latency of requests and thesecond average latency of requests; and computing a number of requestsratio based on the first average number of requests and the secondaverage number of requests.
 8. The system of claim 7, wherein theresizing further comprises increasing a size of the thread poolresponsive to identifying the average number of requests ratio as beinggreater than the average latency ratio and wherein the increasing thesize of the thread pool includes increasing the size of the thread poolby one thread.
 9. The system of claim 7, wherein the resizing furthercomprises decreasing a size of the thread pool responsive to identifyingthe average number of requests ratio as being less than the averagelatency ratio and wherein the decreasing the size of the thread poolincludes decreasing the thread pool by a percentage of the size of thethread pool and wherein the percentage is configurable.
 10. A methodcomprising: executing a job, at a backup host, to back up a file setfrom a source host responsive to a triggering event, the backing up ofthe file set including fetching metadata from the source host;identifying a first operation set from a plurality of operation setsresponsive to identifying a file sharing protocol being utilized by thebackup and the source host, wherein the operation set includes a firstoperation, the identifying being performed by at least one processor;communicating, in parallel, one or more requests, over a network, to thesource host by utilizing one or more threads from a thread pool, thecommunicating the one or more requests including communicating a firstrequest, over the network, to the source host by utilizing a firstthread to fetch a first metadata item; receiving responses, over thenetwork, corresponding to the one or more requests by utilizing the oneor more threads, the receiving the responses including receiving a firstresponse corresponding to the first request by utilizing the firstthread, the first response including the first metadata item; processingthe responses including processing the first response, comprising:generating a first latency based on the first response, and incrementinga number of requests based on the first response; aggregating samplesresponsive to a timeout, the samples including the first latency and thenumber of requests; resizing the thread pool based on the aggregating;and backing up the file set from the source host based on the metadata.11. The method of claim 10, wherein the file sharing protocol is NetworkFile System protocol and the first operation set is a Unix operationset.
 12. The method of claim 10, wherein the file sharing protocol isServer Message Block protocol and the first operation set is a Windowsoperation set.
 13. The method of claim 12, wherein the first requestincludes the first operation, for execution on the source host, togenerate the first metadata item.
 14. The method of claim 10, whereinthe generating the first latency further comprises: identifying areceive time responsive to the first thread receiving the firstresponse; subtracting a transmit time from the receive time to generatethe first latency; and storing the first latency in a first sample in afirst moving average including a first plurality of samples.
 15. Themethod of claim 14, wherein the aggregating the samples furthercomprises: aggregating the first latency over the first plurality ofsamples to generate a first average latency of requests; storing thefirst average latency of requests in a first sample in a second movingaverage including a second plurality of samples; aggregating the numberof requests over the first plurality of samples to generate a firstaverage number of requests; and storing the first average number ofrequests in a first sample in the second moving average including thesecond plurality of samples.
 16. The method of claim 15, wherein theresizing further comprises: computing an average latency ratio based onthe first average latency of requests and the second average latency ofrequests; and computing a number of requests ratio based on the firstaverage number of requests and the second average number of requests.17. The method of claim 16, wherein the resizing further comprisesincreasing a size of the thread pool responsive to identifying theaverage number of requests ratio as being greater than the averagelatency ratio and wherein the increasing the size of the thread poolincludes increasing the size of the thread pool by one thread.
 18. Themethod of claim 16, wherein the resizing further comprises decreasing asize of the thread pool responsive to identifying the average number ofrequests ratio as being less than the average latency ratio and whereinthe decreasing the size of the thread pool includes decreasing thethread pool by a percentage of the size of the thread pool and whereinthe percentage is configurable.
 19. A machine-storage medium and storinga set of instructions that, when executed by a processor, causes amachine to perform operations comprising: executing a job, at a backuphost, to back up a file set from a source host responsive to atriggering event, the backing up of the file set including fetchingmetadata from the source host; identifying a first operation set from aplurality of operation sets responsive to identifying a file sharingprotocol being utilized by the backup and the source host, wherein theoperation set includes a first operation; communicating, in parallel,one or more requests, over a network, to the source host by utilizingone or more threads from a thread pool, the communicating the one ormore requests including communicating a first request, over the network,to the source host by utilizing a first thread to fetch a first metadataitem; receiving responses, over the network, corresponding to the one ormore requests by utilizing the one or more threads, the receiving theresponses including receiving a first response corresponding to thefirst request by utilizing the first thread, the first responseincluding the first metadata item; processing the responses includingprocessing the first response, comprising: generating a first latencybased on the first response, and incrementing a number of requests basedon the first response; aggregating samples responsive to a timeout, thesamples including the first latency and the number of requests; resizingthe thread pool based on the aggregating; and backing up the file setfrom the source host based on the metadata.
 20. The machine-storagemedium of claim 19, wherein the file sharing protocol is Network FileSystem protocol and the first operation set is a Unix operation set.